Exploration

Train separately useful low-rank adapters whose two composition orders disagree on a held-out task. Compare naive merging, sequential application, and ALLORA repair while matching rank and inference cost. Does enforcing the declared square improve joint behavior without erasing individual skills, and how often should the method abstain from composition?

Digital-edition research invitation; not part of the frozen dated PDF.

ch-allora

13 Composable Neural Adapters

ALLORA

Low-rank adapters are attractive because they preserve a frozen backbone while adding modular capabilities. Modularity at the parameter level does not imply modularity in behavior: applying adapter \(i\) before \(j\) can differ from applying \(j\) before \(i\). ALLORA treats that order sensitivity as a declared compositional obstruction and repairs it in the low-rank adapter space.

¹1. ALLORA abbreviates A Lie-Algebraic Low-Rank Adaptation. It preserves separately loadable adapters; it does not replace merging or routing baselines that may be better for a particular deployment. ↩