Finance agents stall near ~50% on hard tasks. How much is the data layer?

Updated November 2026. An earlier version of this post described the June run as MetricDuck added to an agent, said it was reproducible, and led with a single category's result. In that run MetricDuck replaced web search and EDGAR rather than being added to them, each category holds just 3 questions, and the run's records weren't kept. We've corrected the post to say so.

Finding the exact number in the right filing is the slow, error-prone part of financial research for an AI agent. In June 2026 we tested a SEC-data tool on it: a Claude Sonnet 4.6 agent, in Claude Code, ran vals.ai's Finance-Agent v2 questions with MetricDuck instead of web search and EDGAR, and with web search and EDGAR alone, three seeds each. Accuracy came out comparable, and the MetricDuck runs cost $0.77 vs $1.65 per question. SEC figures cite the filing they come from, so you can check the work. Watch a real run or connect it free.

You've probably seen the finance score Anthropic now ships with its models — it's measured on an independent benchmark, vals.ai's Finance-Agent. Version 2 is the hard, current one: the multi-step work a research analyst actually does — pull the exact figures out of 10-Ks, build the model, compare the peers — where even the best models clear only about 46% of questions cleanly. Part of that gap may be the data layer underneath the model. So we held the model fixed (Claude Sonnet 4.6) and swapped its tools: MetricDuck's SEC-data MCP in one arm, web search and EDGAR in the other.

Why would a data tool matter? Because on the hard questions much of the work is finding the data. An agent left to web search and raw EDGAR has to find the right filing, open it, and read the exact figure out of dense HTML, and that's where it can slip: it pulls a number from the wrong period, misses a later restatement, reads a press-release summary instead of the audited statement, or can't show where a figure came from at all. MetricDuck hands the agent the structured number with its source attached.

What the benchmark actually asks

These are real questions from the public Finance-Agent v2 set — the same kind our agent ran. They look simple and turn out not to be: each one hinges on the exact figure, from the right period, in the right filing.

Question (verbatim, abbreviated)	Type	What it takes
"Who is the CFO of Airbnb (NASDAQ: ABNB) as of April 07, 2025?"	Simple retrieval	A point-in-time fact, current as of a date
"How has Netflix's (NFLX) Average Revenue Per Paying User changed from 2019 to 2024?"	Trends	A clean six-year series, one definition
"How many basis points did MU beat or miss its Q3 2024 GAAP gross-margin guidance?"	Beat or miss	Guidance vs. actual, to the basis point
"Of AMZN, META, or GOOG, who plans to spend the most on capex in 2025?"	Complex retrieval	Latest guidance across three filers
"What was the total consideration cost TKO paid for the Endeavor assets…?"	Simple retrieval	One number from a specific transaction
"It is March 11, 2025… will TSM beat or miss Q1 2025 guidance, and by how much? Show your work."	Financial modeling	Build a projection from monthly data

Questions from the vals.ai Finance-Agent v2 public validation set (CC-BY-4.0). Want to watch the agent work one of these end-to-end, with its citations to the filing? See a real run →

The accuracy numbers sit close: many questions ask for a fact the model already knows, so the tools can't change the answer and the arms tie.

By category

vals groups the public questions into nine categories of three questions each, so a category result is 3 questions × 3 seeds — too few to read as a rate. The MetricDuck arm scored higher on some categories, such as Financial Modeling, and lower on others: on Precedents (finding comparable past deals) the web arm won by 33 points. Read them as anecdotes, not rates. See the category table →

Where this fits

Well-funded players are betting that a data layer makes finance agents better. The deep, human-verified, global options are enterprise-priced and sales-gated — right for an institution. MetricDuck is the other end: fully automated, US-SEC-focused, free and self-serve. Not deeper or more accurate than the institutional vendors — the one you can try yourself, right now, at no cost, and check against the filings.

How we measured it — and how to read it

MetricDuck isn't a model; it's an MCP over SEC filings, transcripts, and IR data. So we measured a system: the same model and questions with two tool setups, MetricDuck instead of web search and EDGAR, and web search and EDGAR alone. The gap compares the two setups; it doesn't show what MetricDuck adds to an agent that also has web search. Four things to keep in mind:

• A comparison of two arms, not a leaderboard score. Every number compares the two arms on one fixed model (Claude Sonnet 4.6), against our own free web + EDGAR baseline — so it isn't comparable to vals.ai's hosted ranking.
• Accuracy is comparable, not proven equal. With 27 questions × 3 seeds the accuracy difference wasn't statistically distinguishable — its confidence interval spans 0. We read that as no accuracy difference we could measure, not as proof the two are identical, and not "more accurate" (or "more reliable" — reliability is mixed).
• Checked when it ran. A different-family judge (Gemini) re-graded every answer (κ = 0.92), and the answer is pulled from the agent's response by rule.
• Not re-checkable now. The harness lives in a private repository and the run's records weren't kept, so this run can't be re-derived. Read it as one dated result.

---

See it for yourself: watch a real agent run · the study page.