First-Order Predictable but Pairwise Fragile: Local Task Adaptation in Trained Transformers

This paper evaluates eight local-adaptation properties behind task arithmetic, sequential fine-tuning, activation steering, and first-order random search using the same harness around a multitask LoRA operating point. Across nine Transformers ranging from 82M to 7B parameters, individual perturbation effects remain essentially first-order predictable through the tested scale of 10^-2. Pairwise behavior is much less stable: order sensitivity appears inside that window for more than one-third of model-task pairs, gradient subspaces rotate within tens of steps, activation additivity fails at full task-vector scale on several models, and no model median meets the registered global mean-vector weight-to-steering correspondence threshold.