lin-0169
13.4 Training and repair
ALLORA adapter repair
Train or initialize independently useful low-rank adapters.
Declare which adapter pairs and insertion sites should compose.
Measure pair and multi-adapter order sensitivity on held-out inputs.
Add the layerwise commutator block for declared pairs.
Sweep a registered range of \(\lambda _c\) under task-loss caps.
Retain independently useful adapters and evaluate both application orders.
Admit only Pareto-valid points; compare with merge and routing baselines.
The repair language does not require every adapter pair to commute. Some skills are intentionally order-dependent. The relation set \(\mathcal C\) is part of the application specification.
Regularize only interactions whose order invariance has semantic meaning. Vanishing every bracket can erase useful specialization or force adapters toward trivial behavior.
A Natural LINCS extension could equip the quotient space of adapter realizations with a Fisher or other task-semantic metric. The commutator sketch would still identify which declared interaction requires repair, while the metric would choose the least intrinsic adapter change that contracts it. This would define Natural ALLORA; it would not change the requirement that individual task behavior and both application orders be admitted separately. The experiments below use the declared commutator penalties directly and do not rely on this additional metric structure.