X-AuT: Progressive Audio-Encoder Compression for Speech LLMs via Cross-Scale Distillation
Original title:X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Trimming audio-encoder depth cuts inference overhead in speech LLMs, but directly dropping blocks often perturbs decoder embeddings, triggering token deletions and premature end-of-sequence stops. X-AuT addresses this with a progressive compression framework that selects layer subsets through behavioral probes and recovers performance via cross-scale distillation and LoRA tuning while keeping the language model backbone frozen. Evaluated on Qwen3-ASR-0.6B across ten Chinese-English benchmarks, pruning the audio encoder from 18 to 14 layers reduced audio-tower parameters by 20.7% with a macro-average error of 5.75%, outpacing direct pruning.
Why it's worth reading
As speech LLMs shift toward production serving, X-AuT presents a practical recipe to compress audio towers by over 20% while preventing decoding errors caused by naive layer pruning.