Distill Globally, Adapt Locally: Reasoning Distillation and Product-Type Test-Time Training for Scalable Trade-Up Recommendation
Directly querying large language models across hundreds of millions of product pairs to recommend trade-up alternatives is commercially prohibitive. This paper proposes a two-level architecture that distills RAG-assisted LLM rationales into a compact 15.5M-parameter non-generative classifier. Operating solely on precomputed 768-dimensional embeddings during inference, it further incorporates product-type test-time training (PT-TTT) via lightweight category adapters. Evaluated on an 8,352-pair human benchmark, the system reaches an AUC of 0.941 while cutting estimated inference costs by roughly 10,000x on an eight-GPU machine.
Why it's worth reading
It demonstrates how to package LLM reasoning and test-time adaptation into an ultra-compact, non-generative classifier, offering a practical blueprint for low-latency intent matching at scale.