- Daniel Fried, Churchill. 2015. Low-rank tensor approximations for compositional distributional semantics. Abstract: This thesis explores compositional distributional semantics: methods for mapping words to feature vectors representing their meaning, and composing these word vectors to produce representations of the meanings of longer expressions such as phrases and sentences. Several compositional distributional semantic methods use matrices and their generalization, higher-order tensors, to model multi-way interactions between vectors. Unfortunately, the size of these higher-order tensors has been one obstacle to large-scale implementations of the compositional frameworks that would be able to produce representations for full-length sentences with a diverse vocabulary. In this work, we investigate whether we can match the performance of full matrices and tensors with low-rank approximations that use a fraction of the original number of parameters. We compare low-rank matrices and tensors to full, unconstrained-rank matrices and tensors on standard semantic similarity tasks for two syntactic constructions: adjectives represented by matrices, and transitive verbs represented by third-order tensors. Using lowrank approximations allows us to reduce the number of the parameters in the models by about 40% for matrices, and by 99% (two orders of magnitude) for the third-order tensors. Despite this reduction in the size of the models, the low-rank matrices and tensors achieve performance comparable to, and occasionally surpassing, the full models. The parameters of these low-rank representations can be optimized directly using standard gradient-based methods, allowing them to be incorporated into existing machine learning models for compositional distributional semantics.
