Confidently Wrong, Beautifully Formatted: What Happened When We Made AI Cite Its Tax Research
This week we shipped a research guardrails layer to the OpenAccountants MCP server, and then did the only thing that matters after shipping: we tested whether it actually changes anything.
Short version: on our three-scenario benchmark, the model without guardrails got the substance right 67% of the time. With the guardrails, 100%. But the interesting part isn't the score. It's how the baseline failed.
The failure that should scare you
One of our test scenarios is a US C-corp that spent $900k on software development in 2025: $500k on in-house US engineers, $400k on a contracted dev team in Poland. The question: can we deduct this, or does it have to be capitalized?
This is a deliberately nasty question, because the law changed in 2025. Domestic research costs are immediately deductible again under the new §174A rules; foreign research still has to be amortized over 15 years. Any model trained before the change "knows" the old answer: capitalize everything.
The baseline model, with no guardrails, produced a genuinely beautiful memo. Clear structure. Confidence levels. Citations to the Internal Revenue Code. Our judge scored its citation form at 0.95 out of 1.
It also applied the old law to the whole $900k. Substantive score: 0.10.
That's the failure mode that should worry anyone using AI for tax: not the sloppy answer you'd catch, but the confidently wrong answer wearing a suit. Every citation pointed at a real code section. The memo just described a version of the law that stopped existing months ago.
With the guardrails on, the same model on the same question split the treatment correctly, domestic deductible, foreign amortized, and scored 1.00.
What the guardrails actually are
Here's the design decision that matters: we don't hand the AI tax facts. We're accountants; we know exactly how fast a stored rate goes stale, and who gets blamed when it does.
Instead, when someone asks their AI an analysis-shaped question through our server ("can I deduct this", "how is this taxed", "am I still tax resident"), the response carries a research protocol:
- An authority hierarchy. Statute beats regulation beats case law beats guidance beats commentary, plus the jurisdiction's actual primary sources (the IRS and eCFR for the US, CFR and legislation.mt for Malta, and so on).
- Citation discipline. Every figure anchored to a pinpoint citation, fetched from the official source in this session, never recalled from training data. Stale recall is exactly what produced the §174A failure above.
- A confidence ladder. US questions get the real practice standards (will, should, more likely than not, substantial authority, reasonable basis). Everywhere else, High/Medium/Low with the reason it isn't higher.
- A memo contract. The eight sections a professional reviewer expects, ending with an authorities table and a scope note.
Then the part I like most: before finalizing, the AI calls our verify_citations tool with every claim it wants to make, as {url, figure, quote} triples. Our server fetches each cited page and checks the figure actually appears there. Not against our database, against the primary source itself, live. The AI's confidence stops being self-reported.
And where the position depends on unsettled law or facts nobody can verify, the protocol routes to a named, licensed accountant. Not "consult a professional" as a disclaimer. A specific human, by name, through the same connector.
The numbers
Three scenarios, drawn from real practice: the US §174A question above, an Irish SaaS company selling B2C into Germany (VAT place of supply, the €10,000 threshold, OSS), and a UK professional who moved to Dubai mid-year (statutory residence test, split-year treatment). Same model, same prompts, judged by an LLM judge on three dimensions each.
| Without guardrails | With guardrails | |
|---|---|---|
| Substantively correct, current law | 2 of 3 (avg 0.67) | 3 of 3 (avg 1.00) |
| Citation discipline | strong | strong |
| Confidence and scope discipline | strong | strong |
Two honest observations from the data:
Formatting is not the hard part. Both arms produced well-structured memos with citations and confidence levels, because we asked for them. The gap is currency: whether the beautifully cited law is the law that exists today. That's exactly the gap a fetch-from-primary-sources protocol closes, and a stored-answers database merely relocates.
Models under-use tools. In one of the three with-guardrails runs, the AI answered well but never called the server's front door at all. That's why the protocol lives inside the one call AI assistants already make, rather than a shiny new tool nobody discovers. Distribution of good behaviour is a design problem, not a documentation problem.
The caveats, because we're accountants
Three scenarios is a benchmark, not a study. It was one run per arm, judged by an LLM against rubrics we wrote, on questions we chose. We built the thing being tested. Treat the numbers as a directional result that matched our earlier skill-based benchmarks (100% vs 67% across 18 assertions), not as independent science. The eval suite is part of our release pipeline now, so every change to the server gets re-run against it.
Where this goes
The next layer is research records: when your AI finishes a memo through OpenAccountants, it registers which sources were used, which checks passed, and which named accountant's method backed it, and gets a public verification URL. A reviewer opens the link and sees exactly what stands behind the numbers. "An AI wrote this" becomes "here is the trail."
The method itself is now a public Guide you can read, download, or paste into any AI: Tax research with your AI: how to do it defensibly. And if you want the guardrails running automatically, add OpenAccountants to the AI you already use. And if you're an accountant, the layer above the guardrails is human: build a Guide, put your name on your method, and be the professional the AI hands off to.