OctoLong: Mid-Training on Cross-Repository Code Contexts Enhances Long-Context Modeling

OctoLong is a context-engineering pipeline that combines an AST parser, language-server backend, and package manager to recursively retrieve code references and build dependency-rich contexts reaching millions of tokens. The authors train OctoLong-Instruct models from 600M to 14B parameters using a roughly 50B-token mixture, including about 6.2B tokens of OctoLong code contexts and about 10B tokens of instruction tuning. According to the abstract, replacing only 12% of conventional context-extension data improves long-range retrieval, persistent state tracking, repository-level code understanding, agentic tasks, and short-context API usage.