AdaptiveFedLoRA: Drift-Aware Adaptive LoRA Rank Scheduling for Federated Medical Small Language Models

Abstract

Abstract Federated learning (FL) for medical small language models (SLMs) faces significant challenges due to client drift caused by non-IID clinical data distributions and heterogeneous hardware capabilities. Existing approaches, such as time-based LoRA rank scheduling, fail to adapt to dynamic drift patterns. We propose AdaptiveFedLoRA , a novel drift-aware adaptive LoRA rank scheduling framework that dynamically adjusts model capacity (LoRA rank) based on multi-faceted drift measurements. Unlike prior work that constrains updates (e.g., SCAFFOLD, FedProx), A daptive F ed L o RA dynamically allocates model capacity (via LoRA rank) in response to measured drift. Our approach combines (1) multi-faceted drift measurement (model, performance, and semantic drift), (2) adaptive rank scheduling that responds to drift levels, (3) intelligent client selection, and (4) specialty-aware aggregation using Jensen-Shannon divergence. We validate our method on simulated medical data with Qwen3-0.6B (0.6B parameters) across heterogeneous devices. Eexperimental results demonstrate improved convergence and substantially reduced drift relative to strong FL baselines (FedAvg, FedProx, FedNova, SCAFFOLD, SA-FedLoRA), while maintaining communication efficiency through adaptive parameterization. A daptive F ed L o RA achieves best performance across all tested scales (2-5 clients per round), achieving mean final loss of 0.4982–0.5841 across 2–5 client scales, with 15.5–52.4% improvement over strongest baselines, making it ideal for resource-constrained medical FL deployments. Comprehensive scale experiments (2, 3, and 5 clients per round) reveal scale-dependent performance patterns, with A daptive F ed L o RA maintaining consistent superiority and low variance across all scales, highlighting the importance of scale-dependent method selection. Downstream task evaluation on ICD-10-CM code prediction demonstrates that all methods achieve recall > 0.40 on zero-shot evaluation, confirming that federated learning preserves clinically useful representations that transfer to clinical tasks.

Ähnliche Arbeiten