Lexical Reference Frames: Interpreting LM Residual Streams in Native Semantics
Standard interpretability lenses project transformer residual streams through static directions, often missing context-dependent semantics. This paper introduces Lexical Reference Frames (LexFrame), which interpret intermediate hidden states through a model's own subsequent attention layers rather than an invariant projection. Evaluated on SmolLM3-3B and OLMo3-7B, LexFrame achieves a frame coherence of 0.97 compared to roughly 0.35 for fixed lenses. Beyond faithful attribution patching and targeted token steering, its confident intermediate readings allow dynamic early-exit mechanisms to bypass over 40% of layers during inference.
Why it's worth reading
It replaces static probing lenses with dynamic, layer-aware reference frames, bridging the gap between mechanistic interpretability, localized steering, and practical early-exit inference.