Low-Rank Transformations of the Syntactic-Semantic Space in Multiple Natural Language Tasks

Abstract

This project evaluates the LoRA-multi approach for parameterefficient fine-tuning of Transformer models across various natural language processing tasks. By applying Low-Rank Adaptation (LoRA) to BERT and DistilBERT models and combining them with a Mixture-of-Expert strategy on the X-LoRA framework, we assess their performance on established benchmarks, including SWAG, SQuAD, and WNUT17. Our experimental results validate that while full fine-tuning yields higher performance on specific tasks, the use of LoRA significantly reduces the number of trainable parameters, providing a practical balance between model performance and computational efficiency. This is particularly beneficial for multi-task learning scenarios, where minimizing resource consumption is crucial. Additionally, we explore the implications of these findings for deploying specialized models in real-world applications, highlighting the X-LoRA framework’s capability to maintain versatility while adapting to various downstream tasks. Ultimately, our work aims to advance the understanding of how low-rank adaptations can enhance the efficiency of Transformer models, paving the way to understanding how these models encode syntactic-semantic information.