ARCHead: Activation-Metric Residual Correction for Large Language Model Output Heads
AI Summary
ARCHead compresses the LM output head that weight-only quantization backends often leave in BF16 or FP16. It combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted under an activation-derived metric, without retaining a dense BF16 head. The paper reports 25.6% of BF16 head storage and 1.007 relative perplexity on Qwen3-8B-Base, compared with 1.14-1.16 for storage-matched naive INT4. Replacing heads left by AWQ or bitsandbytes reportedly adds 0.006-0.007 cross-entropy with under 2% throughput change.
Why it's worth reading
The output head can remain a material high-precision storage cost after block quantization, and ARCHead proposes a targeted complement compatible with common AWQ and bitsandbytes workflows.
Deep Read
1. What happened
Original fact: The paper introduces ARCHead, a compressor for the final language-modeling output head. This vocabulary-scale projection is often retained in BF16 or FP16 even when transformer blocks use weight-only quantization. The authors also provide a code repository.
2. Core technology
Original fact: ARCHead combines a quantized low-rank core, group-wise INT4 residuals, and a low-rank correction fitted using a metric derived from activations. It stores no dense BF16 head. The method is presented as a complement to transformer-block quantizers rather than a replacement for them.
3. Key evidence and numbers
Original facts, all reported in the abstract: Persistent LM-head storage falls by 3.7-3.9x. On Qwen3-8B-Base, ARCHead uses 25.6% of BF16 head storage with 1.007 relative perplexity, while storage-matched naive INT4 reaches 1.14-1.16. Replacing the BF16 head retained by AWQ or bitsandbytes adds 0.006-0.007 cross-entropy, with less than 2% throughput change in the authors' measurements.
4. Why it matters
Analysis: Once transformer blocks are compressed to low precision, the output projection can become a more visible share of remaining storage because it scales with hidden width and vocabulary size. If reproducible, ARCHead could close this end-to-end compression gap while preserving the vocabulary-logit distribution better than direct INT4 quantization.
5. Practical impact
Analysis: The approach is most relevant to deployments with large output heads, existing AWQ or bitsandbytes pipelines, and tight limits on loading memory, distribution size, or persistent weights. The reported throughput delta suggests modest runtime overhead, but realized gains will depend on backend support for the packed representation and correction path.
6. Limitations and uncertainty
Original fact: The supplied abstract gives concrete results only for Qwen3-8B-Base and does not specify broader model families, vocabulary sizes, downstream tasks, hardware configurations, or sustained-serving tests. Analysis: Near-neutral perplexity does not by itself establish unchanged generation quality, rare-token ranking, or downstream accuracy. Unverified: The supplied publication date is 2026-08-02 and is future-dated metadata; the reported results have not been independently reproduced or verified here.
7. Original sources
- Paper abstract: https://arxiv.org/abs/2608.02703
- Code repository: https://github.com/suayptalha/archead