Efficient Fine-Tuning Strategies for Domainspecific Large Language Models

Abstract

The developing computational fee of excellenttuning large language models (LLMs) for area-particular packages poses a sizable challenge, necessitating parametergreen strategies that preserve performance at the same time as lowering useful resource needs. This takes a look at evaluates 3 fine-tuning techniques-Low-Rank Adaptation (LoRA), Adapter Layers, and Prompt-Tuning-towards conventional full exceptional-tuning using XLM-RoBERTa skilled on multilingual criminal (CUAD) and clinical (MIMIC-III) datasets. Our experiments show that LoRA achieves 91.5% accuracy (vs. 92.3 % for complete satisfactory-tuning) with a ninety nine% reduction in trainable parameters, offering nearidentical overall performance to complete high-quality-tuning even as notably decreasing computational fees. Adapters show sturdy generalization in criminal textual content class (F1score: 90.5 %), while activate-tuning, although the maximum green (0.1 M parameters), lags in complicated medical obligations (accuracy: <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{8 8. 2 \%}$</tex>, perplexity: 18.7). This painting offers the first complete comparison of parameter-efficient methods in multilingual prison and medical domains, revealing crucial project-particular trade-offs: LoRA excels in factsmoderate eventualities, adapters suit specialized terminology edition, and prompt-tuning aligns with low-aid constraints. The study underscores the need to align exceptional-tuning techniques with area complexity and resource availability. Future paintings must explore hybrid techniques (e.G., LoRA Adapters) to optimize both performance and flexibility, alongside scaling opinions to large LLMs (e.G., 70B parameters).

Ähnliche Arbeiten