OctoLong: Mid-Training on Cross-Repository Code Contexts Enhances Long-Context Modeling
Original title:OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling
OctoLong is a context-engineering pipeline that combines an AST parser, language-server backend, and package manager to recursively retrieve code references and build dependency-rich contexts reaching millions of tokens. The authors train OctoLong-Instruct models from 600M to 14B parameters using a roughly 50B-token mixture, including about 6.2B tokens of OctoLong code contexts and about 10B tokens of instruction tuning. According to the abstract, replacing only 12% of conventional context-extension data improves long-range retrieval, persistent state tracking, repository-level code understanding, agentic tasks, and short-context API usage.
Why it's worth reading
Long-context progress is shifting from merely enlarging the window to engineering dependency-dense data. This paper offers a concrete recipe for training code-capable models and agents across repository boundaries.