PhD Thesis:Discovering semantic relations using singular value decomposition based techniques
I did my PhD thesis in 2009 at Brandeis University under James Pustejovsky. I was part of a group of people in the great field of common sense who were very early to the concept of word embeddings which became extremely popular with Google's Word2Vec and Deep Learning. Very early on I recognized that the amount of data to train these systems was outside of what was accessible for language use cases and started experimenting with domain adaptation and transfer learning. The colloquial name for this algorithm was "blending". The general approaches we used tended to be different since we were working in this field in the pre-GPU era. Further experimentation of this with many students at the MIT Media Lab led to several projects and masters's theses before it became the company Luminoso.
Here's the abstract:
Understanding the world we live in requires access to a large amount of background knowledge: the common sense knowledge that most people know and most computer systems don't. The inability to acquire and understand semantic information, especially common sense knowledge, has constrained current artificial intelligence systems.
Much progress has been made in manually acquiring this knowledge, both in the form of lexical resources created by trained lexicographers and in those resources created by using information obtained from volunteers on the Internet. Reducing the dimensionality of this knowledge using singular value decomposition (SVD) yields a matrix representation called AnalogySpace, which reveals large-scale patterns in the data, smooths over noise, and predicts new knowledge.
Extending this work, I have created a method that uses singular value decomposition to aid in the integration of systems or representations. This technique, called blending, can be harnessed to find and exploit correlations between different resources, enabling common sense reasoning over a broader domain.
The power in blending is its ability to combine knowledge sources and to inject broad semantic and common sense knowledge into other data sets and applications. I evaluate the performance of blending in several scenarios and discuss potential applications of blending to many communities.
There was also a journal article for those looking for a shorter read.