Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA

This paper studies baking documents into LoRA adapters for closed-book question answering with a 4-bit Gemma-4-e4b model, eliminating retrieval and context-window usage at inference time. Across roughly 100 runs covering one to 99 documents, the authors report that data quality dominates rank, learning rate, and two alternative architectures once adapter capacity is sufficient. On a 15-document corpus, shortening gold answers to canonical 1–6-word spans and removing trivia increased accuracy from 57.7% to 85.7%. The internalized adapter reached 84.2% recall, compared with 58.9% for BM25-RAG and 65.6% for a gold-chunk oracle baseline.