Why Enterprise AI Stumbles on 233 Years of Treasury Data

Databricks Drops a Harder Enterprise Reasoning Benchmark Built on 233 Years of Treasury Data

Databricks has released OfficeQA Pro V2, a new benchmark that tests whether AI agents can truly generalize their grounded-reasoning abilities to unfamiliar enterprise document collections. The 90-question evaluation set is built atop roughly 120,000 pages of U.S. Treasury records spanning from 1793 through 2024 — a corpus the Treasury released as a unified dataset for the first time in conjunction with America’s 250th anniversary.

Out-of-the-box frontier agents stumbled hard, averaging just 26.0% accuracy when using their default model-provider harnesses. Yet the same underlying models, when routed through Databricks Genie — the company’s own agent framework with pre-parsing via its ai_parse tool — jumped to an average of 52.8% accuracy across matched comparisons, a 92% relative improvement that still leaves substantial headroom unsolved.

The benchmark originally served as the evaluation for the inaugural Databricks Grounded Reasoning Cup, where 11 academic teams backed by OpenAI, Anthropic, and Google DeepMind competed. The winning entry reached 63.3%, setting the current ceiling on a test that demands retrieval across an average of 6.7 source documents per question.

Synthetic Data Pipelines Replaced Manual Annotation to Scale Grounded-Reasoning Tests

When Databricks built the original OfficeQA benchmark in late 2025, human annotators constructed questions by hand while combing through 89,000 pages of U.S. Treasury Bulletins. The process was painstaking, expensive, and resisted scaling.

As Databricks detailed, OfficeQA Pro V2 took a radically different path. The engineering team deployed asynth, an internal synthetic data-generation library, to automate question creation through a multi-stage pipeline.

Each candidate question began with a randomly seeded combination of a time period, an analytical method, a target number of source documents, and a salient topic drawn from a list developed with partner USAFacts. A synthesis agent then searched the corpus for evidence matching those criteria and generated a sample backed by a traceable reasoning chain.

From there, samples passed through quality gates auditing source fidelity, analytical relevance, diversity, and answerability. Independent solver agents attempted each question from scratch, producing alternate solution trajectories reviewed by a verification agent. Only candidates with verifiably correct answers advanced. As a final step, human reviewers examined questions alongside solutions from additional frontier models, retaining only samples where interpretation and ground truth were unambiguously correct.

The resulting 90 questions preserve the same core grounded-reasoning capabilities measured by the original OfficeQA: retrieval across multiple source documents, analytical reasoning, and specialized demands like supplemental web search and multimodal interpretation of figures. Roughly 7% of questions require visual understanding of charts or graphs, while 10% require external data such as inflation indices or GDP series.

Multi-source retrieval dominates the difficulty profile. Questions draw on an average of 6.7 source documents with a median of 5.5 and a maximum of 24. In total, 74.4% of questions require four or more sources — a heavier lift than OfficeQA Pro’s 62.4%.

Two Centuries of Shifting Reporting Standards

The corpus itself is a stress test in document heterogeneity. Records from the late 1700s feature archaic typography — the long-s character appearing in words like ‘Treaſury’ or ‘Preſident’ — alongside wide landscape foldouts and ‘ditto’ shorthand that modern parsers struggle to interpret correctly.

By the mid-19th century, data shifted to two-page T-account formats and dense ruled tables, while reporting periods migrated from calendar to fiscal years. The early 1900s introduced opening and closing balances, multi-year appropriations, and surplus and deficit conventions.

By the 1980s, the Combined Statement had evolved into an annual report with detailed appendices. Digital-native reports arrived in the early 2000s, incorporating visual charts alongside standardized tables of contents.

Across the full 232-year span, the same financial concept may change in name, location, table schema, unit, time basis, and aggregation level. Agents must reconcile information across shifting reporting conventions — precisely the kind of complexity that defines real enterprise environments.

Frontier Agents Keep Stumbling — and a Rival Benchmark Exposes Deeper Gaps

The benchmark’s difficulty is not an accident of obscure document selection. It reflects structural challenges that persist even in the strongest current systems, as corroborated by independent evaluations from major AI labs and academic researchers.

Anthropic’s Claude Opus 5 system card, published in mid-2026, dedicates a full section to evaluating the model on the original OfficeQA and OfficeQA Pro benchmarks. Claude Opus 5 achieved 66.9% on OfficeQA Pro — solid performance on the 133-question frontier subset — but that test draws on an average of roughly 2 Treasury Bulletin documents per question. OfficeQA Pro V2 more than triples that retrieval demand.

The gap between the original benchmark and V2 underscores a core question Databricks set out to answer: do improvements on OfficeQA reflect genuine advances in grounded reasoning, or progress specific to one corpus and task distribution? Early evidence from V2 suggests the latter remains an open problem.

Meanwhile, a separate research paper introducing FinanceComplexQA — a bilingual benchmark for agentic reasoning over financial documents published on arXiv — explicitly compares itself against OfficeQA-Pro and flags two structural limitations. The original benchmark does not target open-ended answering, nor does it support cross-layout evidence aggregation, both marked as unsupported in the paper’s comparison table.

FinanceComplexQA positions itself as filling those gaps, supporting open-ended answers, cross-layout reasoning, and bilingual tasks across more complex financial scenarios. This competitive pressure highlights an evolving landscape where no single benchmark fully captures the range of enterprise grounded-reasoning demands.

Smaller model builders are also engaging with the OfficeQA suite. The Nanbeige4.2-3B technical report evaluates its compact 3-billion-parameter model on OfficeQA-Pro, achieving 21.1% — notably higher than Qwen3.5-9B’s 15.8% and dramatically above Gemma4-E4B’s 3.1%. These results confirm the benchmark’s difficulty gradient and its growing adoption as a standard evaluation tool across the open-source model ecosystem.

Parsing fidelity, incorrect temporal reconciliation as accounting conventions shift, and misinterpretation of entity scope or category granularity remain persistent failure modes. Even Genie’s strongest configuration, reaching 60%, leaves 40% of questions unsolved — a gap that no amount of harness optimization alone appears able to close.

What 60 Percent Accuracy Tells Us About the Real State of Enterprise AI

OfficeQA Pro V2 lands at a moment when enterprise adoption of AI agents is accelerating, yet the data makes one reality difficult to ignore: the best publicly documented system on this benchmark still fails on four out of every ten questions. Those failures are not random noise but concentrated in precisely the areas — multi-document reconciliation, temporal reasoning across changing conventions, faithful parsing of dense tables — that enterprise deployments encounter daily.

The benchmark and parsed corpus are publicly available on Hugging Face, with evaluation code on GitHub, giving enterprise developers a testbed to compare models and agent architectures against their own data challenges before committing to production architectures. Teams can isolate whether failures originate in parsing, retrieval, reasoning, or verification — and measure how changes to their systems affect accuracy, latency, and cost.

For organizations building AI agents that must reason over large, heterogeneous document collections, the implications are clear: model quality alone is insufficient, harness design creates massive leverage, and representative evaluation on unfamiliar corpora is the only reliable signal of generalization. At scale, AI automation and programmatic workflows become the backbone of reliable agent deployment, much as robust cloud infrastructure underpins every production system that cannot afford downtime. To explore how these capabilities translate to measurable performance gains, connect with Andres directly or learn more about Andres SEO Expert and the technical approach behind the results.