Are Financial Reasoning Capabilities of LLMs Credible? A Real-World Test over Long-Horizon Statements
Original title:Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements
AI Summary
The paper introduces FinIndices, a benchmark built from uncropped financial statements of up to 32K tokens. It evaluates single-index computation and multi-table index tabulation, including cross-statement, temporal, and stock-flow reasoning. According to the abstract, removing explicit formula hints reduces Gemini-3.1-Pro’s table-task performance from 70.70% to 38.22%, indicating fragile pattern matching and failures in temporal de-cumulation and accounting caliber alignment. Multi-metric, multi-period output also causes models to fall back to shallow heuristics. Supervised fine-tuning improves zero-hint performance by 8.54% on Single-Index and 3.82% on Table-Index tasks.
Why it's worth reading
Long financial statements expose failure modes that ordinary QA benchmarks miss. FinIndices is timely because it tests whether models actually align periods, statements, and accounting definitions before they are trusted in financial workflows.
Deep Read
What happened
Original facts: The paper proposes FinIndices, a benchmark for testing LLM data-processing fidelity on uncropped financial statements, with contexts of up to 32K tokens. It includes Single-Index computation and Table-Index tabulation tasks.
Core technology
Original facts: FinIndices uses an automated synthesis pipeline with adversarial traps. The benchmark targets domain reasoning, temporal reasoning, and accounting-caliber reasoning, including temporal de-cumulation and stock-flow mismatches.
Key evidence and numbers
Original facts: Without explicit formula hints, Gemini-3.1-Pro reportedly falls from 70.70% to 38.22% on table tasks. SFT improves zero-hint performance by 8.54% on Single-Index and 3.82% on Table-Index tasks. The abstract does not report the full model roster, confidence intervals, or error distributions.
Why it matters
Analysis: The benchmark separates memorizing formulas from correctly locating values, aligning periods, and preserving accounting definitions across statements. If the reported findings generalize, short-table QA benchmarks may overestimate reliability in financial analysis.
Practical impact
Analysis: Production systems should expose auditable intermediate steps for statement extraction, period conversion, stock-flow checks, and formula execution. Multi-metric and multi-period outputs need item-level validation rather than reliance on a polished final answer. SFT may improve robustness, but it does not replace provenance tracking or deterministic checks.
Limitations and uncertainty
Original facts: The benchmark relies on automated synthesis and adversarial traps. The abstract does not specify the proportion of real-company statements, annotation audits, industry coverage, or contamination checks. Unverified inference: If the task templates or trap distribution differ materially from real workflows, the measured degradation may not transfer directly to every financial application.
Original sources
- arXiv abstract and paper page
- Source label: hf-papers; publication date: 2026-07-21