AI agents handle easy finance lookups fine — and stall on the hard, multi-step work: pulling the exact figure, from the right period, in the right filing. That bottleneck is the data layer, not the model. So we measured it — the same agent, the same questions, a SEC-data tool toggled on and off.
Same model (Claude Sonnet 4.6), MetricDuck toggled on vs off. Overall accuracy comes out even — the durable win is efficiency and the hard stuff; we don't claim "more accurate."
On the harder Finance-Agent v2, even the best models clear only ~46% of questions cleanly (all-pass) and ~58% with partial credit — multi-step financial analysis is genuinely hard. See the vals.ai Finance-Agent v2 benchmark ↗ — the independent benchmark we reproduce. Below is the same agent, with and without MetricDuck.
| Metric | Without (web + EDGAR) | With MetricDuck | Difference |
|---|---|---|---|
| Accuracy (all-pass) | 53% | 57% | ≈ same |
| Cost per test | $1.65 | $0.77 | 2.1× cheaper |
| Latency | 495s | 237s | 2.1× faster |
| Agent steps† | 88 | 28 | fewer |
Same model (Claude Sonnet 4.6), same 27 questions from the vals.ai Finance-Agent v2 set, the MCP toggled on and off (N=3 seeds). Accuracy is comparable — its lift confidence interval spans 0, so we say same, not higher. Cost per correct answer: $0.60 with vs $1.21 without.
† “Agent steps” is a harness-side iteration count we're still re-confirming (it conflates the agent's text and tool-call events) — read cost and latency as the robust efficiency signals.
Accuracy delta (with − without): +3.7pp, 95% CI [−11.4pp, +18.6pp] — spans 0 at this sample size (within noise). Excluding the 7 questions whose published gold answer we flag as disputed: +3.3pp — also within noise. We report both; neither is significant.
The robust result is efficiency. Cost per correct answer: $0.60 with MetricDuck vs $1.21 without — about half. The agent reaches answers at ~½ the latency (237s vs 495s median) and takes fewer steps† to get there, and gives up about half as often (no-usable-answer rate 4.9% vs 9.9%). Accuracy is comparable — see the delta above and the per-category cut below.
vals' single hardest category, Financial Modeling, is building a company's valuation from its filings — pulling the exact restated, period, and segment figures and assembling them. A free agent gets 11% right; the same agent with MetricDuck gets 56%. The pattern below: MetricDuck pulls ahead where the work is assembling many exact figures, and ties on easy questions.
| vals category | With MetricDuck | Without (web + EDGAR) | Lift |
|---|---|---|---|
| Financial Modeling | 56% | 11% | +44pp |
| Disclosure Analysis | 33% | 11% | +22pp |
| Comparables | 22% | 11% | +11pp |
| Adjustments | 33% | 22% | +11pp |
| General Quantitative | 67% | 67% | 0 |
| General Qualitative | 56% | 56% | 0 |
| Market Analysis | 89% | 100% | -11pp |
| Earnings Analysis | 89% | 100% | -11pp |
| Precedents | 67% | 100% | -33pp |
We disclose the other side too: on Precedents (finding comparable past deals) the free agent wins by 33 points — MetricDuck splits vals' two hardest categories. The ±11pp rows are seed noise on already-easy categories.
MetricDuck's own curated cross-corpus tasks (transcripts, 8-K, IR, FPI) — illustrative of value, not a neutral benchmark.
| Arm | Accuracy | Cost / query* | Agent steps† |
|---|---|---|---|
| Without MetricDuck (web + EDGAR) | 71% | $0.54 | 31 |
| With MetricDuck | 100% | $0.25 | 12 |
Accuracy delta (with − without): +0.29, 95% CI [+0.049, +0.531].
*Cost is list-price-equivalent (token cost at published rates) — a relative efficiency indicator. Numbers are reported per set and never blended.
† “Agent steps” is a harness-side iteration count we're still re-confirming (it conflates the agent's text and tool-call events) — read cost and latency as the robust efficiency signals.
Want to see it, not just the numbers? See agents at work → (every figure clicks back to the SEC filing it came from).
The data layer makes finance agents better — a category now validated by well-funded players. The enterprise options are paid, sales-gated, and built for institutions. MetricDuck is the free, self-serve, US-SEC-focused one you can verify yourself before connecting.
MetricDuck is the data layer beneath whatever agent you run — not another model. The numbers below are the same agent, with the MCP toggled on and off.
We reproduced the methodology of the vals.ai Finance-Agent v2 benchmark (the harder, multi-step version the current finance-agent discourse runs on) and ran the same agent two ways on identical questions: without MetricDuck (free web + SEC EDGAR tools, the vals-comparable baseline) and with the MetricDuck MCP. The difference between the two arms isolates MetricDuck's contribution.
It's a public benchmark, so memorization is a fair worry — and the with-vs-without design controls for it. Both arms run the same model on the same questions; the MetricDuck MCP is the only thing that changes. Any memorized answer helps both arms equally, so the lift and efficiency gains live in the delta, not the absolute score — they can't be explained by contamination.
To rule out a model grading its own family favourably, we re-graded with a different-family judge (Google Gemini) on identical inputs and rubric. Agreement was "almost perfect" (Cohen's κ = 0.918) — no meaningful self-preference bias.
The answer is extracted from the agent's committed response by rule (not re-interpreted by an LLM), and the judge never sees which arm produced an answer.
Every run records the harness version, the MCP version, the data snapshot, and the seed, so a result can be re-derived.
The robust signal is efficiency (above). On accuracy, the N=3 aggregate lift is +3.7pp with a 95% CI that spans 0 — MetricDuck matches a strong free agent's accuracy at about half the cost. We don't claim "more accurate"; the durable edge is cost and the hard-category capability (the by-category cut above).
Many public-benchmark questions concern well-known facts the model already knows, so a tool can't change the answer. There, read the baseline calibration and the efficiency delta, not the absolute accuracy.
The realistic-workflow results come from tasks we authored to represent real analyst work. They're illustrative of where MetricDuck helps — not an independent, neutral benchmark, and we label them as such.
This is our own reproduction of the vals.ai Finance-Agent v2 methodology (the current, harder version), not the official vals.ai leaderboard. Our scores aren't directly comparable to their published numbers — the apples-to-apples comparison is our with-MetricDuck arm vs our without arm, inside one fixed stack.
The without-MetricDuck arm uses only free web + SEC EDGAR tools and never touches MetricDuck, so it's a fixed control we carry forward across runs. Every figure here is a with-vs-without delta on one model (Sonnet 4.6) vs that same baseline.
MetricDuck fails catastrophically about half as often (no usable answer 4.9% vs 9.9%), but the free agent's correct answers are slightly more repeatable across seeds. We report both, and never claim blanket reliability superiority.