In June 2026 we ran Claude Sonnet 4.6, in Claude Code, on the 27 public questions of vals.ai's Finance-Agent v2, three times each, in two setups: with MetricDuck instead of web search and SEC EDGAR, and with web search and SEC EDGAR alone. It compares the two tool setups. It doesn't measure an agent with MetricDuck added.
27 FABv2 questions, Sonnet 4.6, June 2026: $0.77 vs $1.65 per question, accuracy comparable.
A dated study. The harness lives in a private repository and the June run's records weren't kept, so these figures can't be re-derived or independently re-checked. Read them as one dated run. We don't claim "more accurate".
On the harder Finance-Agent v2, even the best models clear only ~46% of questions cleanly (all-pass) and ~58% with partial credit — multi-step financial analysis is genuinely hard. See the vals.ai Finance-Agent v2 benchmark ↗ — the independent benchmark whose public questions we used. Below are the two arms: one model with MetricDuck instead of web search + EDGAR, and with web search + EDGAR alone.
| Metric | Web search + EDGAR | MetricDuck instead |
|---|---|---|
| Accuracy (all-pass) | 53% | 57% |
| Cost per question | $1.65 | $0.77 |
June 2026: Claude Sonnet 4.6 in Claude Code on the 27 public questions of vals.ai Finance-Agent v2, 3 seeds, with MetricDuck instead of web search + EDGAR in one arm. Accuracy is comparable: the difference's confidence interval spans 0, so we say same, not higher.
Accuracy difference (MetricDuck arm − web arm): +3.7pp, 95% CI [−11.4pp, +18.6pp] — spans 0 at this sample size (within noise). Excluding the 7 questions whose published gold answer we flag as disputed: +3.3pp — also within noise. We report both; neither is significant.
The result: 27 FABv2 questions, Sonnet 4.6, June 2026: $0.77 vs $1.65 per question, accuracy comparable. It compares two tool setups for one model; it doesn't measure what MetricDuck adds to an agent that also has web search.
vals groups its public questions into nine categories of 3 questions each, so every row below is 3 questions × 3 seeds: too few to read as a rate. Read the rows as anecdotes, not as where MetricDuck helps.
| vals category (3 questions) | MetricDuck instead | Web search + EDGAR | Difference |
|---|---|---|---|
| Financial Modeling | 56% | 11% | +44pp |
| Disclosure Analysis | 33% | 11% | +22pp |
| Comparables | 22% | 11% | +11pp |
| Adjustments | 33% | 22% | +11pp |
| General Quantitative | 67% | 67% | 0 |
| General Qualitative | 56% | 56% | 0 |
| Market Analysis | 89% | 100% | -11pp |
| Earnings Analysis | 89% | 100% | -11pp |
| Precedents | 67% | 100% | -33pp |
On Precedents (finding comparable past deals) the web arm won by 33 points. With 3 questions × 3 seeds per row, an 11pp difference is one answer in nine.
Cost is list-price-equivalent (token cost at published rates) — a relative indicator.
Want to see it, not just the numbers? See agents at work → (SEC figures cite the filing they come from).
Well-funded players are betting that a data layer makes finance agents better. The enterprise options are paid, sales-gated, and built for institutions. MetricDuck is the free, self-serve, US-SEC-focused one you can check yourself before connecting.
MetricDuck is the data layer beneath whatever agent you run — not another model. The numbers below compare one model with two tool setups: MetricDuck instead of web search + EDGAR, and web search + EDGAR alone.
We ran our own harness on the public questions and methodology of the vals.ai Finance-Agent v2 benchmark (the harder, multi-step version the current finance-agent discourse runs on) and ran one model two ways on identical questions: with free web search + SEC EDGAR tools (the vals-comparable baseline), and with the MetricDuck MCP instead. The difference between the arms compares the two tool setups; it doesn't measure what MetricDuck adds to an agent that also has web search.
To rule out a model grading its own family favourably, we re-graded with a different-family judge (Google Gemini) on identical inputs and rubric. Agreement was "almost perfect" (Cohen's κ = 0.918) — no meaningful self-preference bias.
The answer is extracted from the agent's committed response by rule (not re-interpreted by an LLM), and the judge never sees which arm produced an answer.
The harness lives in a private repository and the June run's records weren't kept, so these figures can't be re-derived or independently re-checked. Read them as one dated run.
The MetricDuck arm had MetricDuck instead of web search + EDGAR. The run shows how the two tool setups compare; it doesn't show what MetricDuck adds to an agent that keeps web search.
It's a public benchmark, so memorization is a fair worry. Both arms ran the same model on the same questions, but the tools differed, so a memorized answer needn't help both arms equally: it can lift the absolute scores and can also shift the difference between them.
The N=3 aggregate difference is +3.7pp with a 95% CI that spans 0, so the arms are comparable. We don't claim "more accurate".
Many public-benchmark questions concern well-known facts the model already knows, so a tool can't change the answer. There, read the baseline calibration and the difference between arms, not the absolute accuracy.
We ran the vals.ai Finance-Agent v2 methodology (the current, harder version) in our own harness, not on the official vals.ai leaderboard. Our scores aren't directly comparable to their published numbers — the apples-to-apples comparison is our MetricDuck arm vs our web + EDGAR arm, inside one fixed stack.
The web arm uses only free web + SEC EDGAR tools and never touches MetricDuck. Every figure here compares the two arms on one model (Sonnet 4.6).
In this run the MetricDuck arm failed catastrophically about half as often (no usable answer 4.9% vs 9.9%), but the web arm's correct answers were slightly more repeatable across seeds. We report both, and never claim blanket reliability superiority.