CDCO: Cross-Domain Contrastive Optimization Framework for Enhancing Multi-Task Learning in Small Pre-trained Language Models

Abstract

The adapter technique has significantly improved the performance of fine-tuning pre-trained language models (PLMs) in multi-task settings, especially for small models with limited resources. However, current adapter frameworks typically rely on single-task data, a fixed representation space, and a single training phase. This approach limits their ability to generalize across domains in multi-task scenarios, thus restricting performance improvements for smaller models. To address these challenges, we propose the cross-domain contrastive optimization (CDCO) framework to enhance performance in multi-task learning. CDCO improves model performance by asynchronously co-optimizing across diverse task data sources, representation spaces, and multi-stage structures. Specifically, CDCO introduces innovations in both data sample selection and training strategy. First, CDCO introduces out-of-domain manifold sampling (ODMS), which enhances training diversity by selecting challenging hard-negative samples from out-of-domain datasets through manifold learning. Second, CDCO employs multi-stage asynchronous co-optimization (MAC), mapping samples from ODMS to Euclidean and Poincaré spaces. Then, it constructs a cross-domain contrastive loss based on the spatial properties of these distributions to guide the optimization process. By sequentially optimizing adapter layers across different spatial distributions, CDCO maximizes the potential of the adapter while mitigating overfitting, thus improving the adaptability and stability of small models in multi-task environments. Experimental results demonstrate that CDCO significantly improves performance on in-domain (ID), out-of-domain (OOD), and knowledge-intensive (KI) tasks, confirming its broad applicability and effectiveness.

Ähnliche Arbeiten