How accurate is AI on tax?

SummaryThe experiment, and the five findings

We gave AI agents one instruction — write the tax law of every country on earth, six topics each, every claim carrying its source — and let them run for three months. Out came 1,928 guides across 186 countries, containing 66,253 individually checkable facts. Fourteen accountants then marked 2,713 of those facts in five languages. Section 2 sets out exactly how. Everything below rests on that, plus a benchmark we built from the marked answers so the same questions can be put to any model.

Section 1What we measured, and what we cannot

Two separate things are measured here and they must never be quoted as one. The first is how good our own drafts were, judged by the accountants who reviewed them. The second is how good AI is at tax generally, measured by sitting a model down in front of a benchmark. The first is a floor. The second is a ceiling. Neither is "the accuracy of AI on tax", and anyone offering you that single number is selling something.

Why our own number is a floor. It counts how often an accountant flagged one of our drafted facts. That is not how often the draft was wrong. An error nobody noticed looks exactly like a fact that was right. The true error rate is that number or higher, never lower.

That distinction is not theoretical, and the fairness point cuts toward the reviewers. They were told, in writing: "Do I need to read every section? No. Review the sections you are confident about. Flag or skip what you are unsure of." and "You're not the last line of defence." A reviewer who confirmed everything was following the instructions we gave them. Where this report finds thin review, that is a finding about our instrument, not about anyone's diligence.

Why the benchmark number is a ceiling. We made the questions easy on purpose, and we drew them from a pool that is biased in the model's favour. Section 15 sets both problems out in full. The short version: treat 77% and 88% as the best case, not the expected case.

Section 2What we actually did

In April 2026 we asked AI to write the tax law of the world. Six topics for every country on earth. Then we asked qualified accountants to mark it. That is the experiment this report is built on, and as far as we can tell nobody has run it at this scale before, so the method matters as much as the result.

Step 1 — the instruction. One web-researching agent per country, each asked to produce the same six guides:

Step 2 — the required output shape. The agents could not write prose and stop. Every claim had to come back as a structured fact with its own fields — the item, the value, the unit, any qualifier, what kind of thing it is (rate, threshold, rule, formula, definition), the legal reference, whether that reference had actually been researched, a source URL, and an effective date.

That constraint is the reason this report is possible. A guide written as an essay cannot be marked fact by fact; 66,253 individually addressable facts can. It is also what let us later discover that 100% of drafted facts carried a legal reference and 72.7% carried a live source URL at the moment they were written — the agents really were reading sources, not reciting.

Step 3 — what came out.

April went wide — 180 jurisdictions in one pass, a thin layer over most of the world. May and June went deep, adding topics to countries that already had one.

Step 4 — the marking. We sent accountants a workbook: one row per fact, with the drafted value, a verdict column, a corrected-value column and a notes column. Twenty-five workbooks came back, in five different layouts and five languages — English, Portuguese, Spanish, Georgian and Indonesian — because the reviewers were working accountants in their own countries, not annotators we trained. Each verdict was loaded back against the exact fact it judged, with the workbook's SHA-256 recorded so any row can be traced to the file it came from.

What the reviewers were told matters for reading every number that follows: "Review the sections you are confident about. Flag or skip what you are unsure of." They were not asked to audit exhaustively, and they were told explicitly they were not the last line of defence.

Section 3The benchmark

As of August 2026 we could find no public benchmark measuring whether AI-stated tax rules and thresholds are correct across multiple jurisdictions. Every tax benchmark we located is single-jurisdiction: TaxCalcBench and SARA are US, TaxPraBen is China, FiscalQA Pro is France, SteuerEx is Germany, RO-FIN-LLM is Romania. The two cross-jurisdictional legal benchmarks we found, Multi-Legal-Bench and CrossLex, exclude tax entirely. One multi-jurisdiction tax benchmark has been announced — TaxOS's TaxBench v1, due Q1 2026 — and has not been released.

That is a claim about a search rather than about the world, and we would rather be shown a counter-example than assert a first. But it is why we built our own, from material we have and others do not: tax facts a named, credentialled accountant has personally adjudicated.

OA-TAXFACTS-60. Sixty questions drawn from facts reviewed by accountants between May and July 2026, capped at six per country so no single reviewer's workbook could define the result. Fifteen jurisdictions: India, Nigeria, Venezuela, Cameroon, Pakistan, Cyprus, Tanzania, Peru, Nepal, Portugal, South Africa, Indonesia, Canada, Saudi Arabia and Brazil.

Each question gives the model the jurisdiction, the guide and section the fact sits in, the item being asked about, the governing statute, and the period the answer must be in force for. That is far more help than a real user gives. It is deliberate: we wanted to know what the model can do at its best, because a failure under generous conditions is a much stronger finding than a failure under hard ones.

The controls, because a benchmark you cannot trust is worse than none.

• The answer key was quarantined. Questions and answers were written to separate directories. The model runs read only the questions.
• The searching arm was barred from us. It could not use openaccountants.com or any of our tools. Our site publishes the very answers being scored; letting it read them would have graded our data against itself.
• Grading was blind and doubled. Two independent graders marked the same sheet with the arm labels stripped and the rows shuffled, so neither could tell which answer came from the searching run. They agreed on 52 of 54 judgement calls (96%). The two disagreements were both "partially right" versus "right", and we took the stricter mark in each.
• Exact matches never reached a grader. 66 of 120 answers were settled mechanically. Nobody exercises judgement over whether "15%" equals "15.0%".
• Wrong year counts as wrong. Being right about 2024 when asked about 2026 is the failure this study exists to measure.

Section 4The scoreboard

Because both arms answered the identical questions, we can compare them question by question rather than only in aggregate — a much stronger test than two separate scores.

Seven fixed and none broken. On a paired test that is significant (McNemar exact, p = 0.016), which in plain terms means it is unlikely to be luck. Search does not trade one error for another. It removes a specific class of error and adds nothing back.

The confidence intervals overlap, and at sixty questions they were always going to. The paired comparison is what carries the finding; the two headline percentages are the context around it.

Section 5What search fixes: facts that went stale

All seven repairs are the same story. The model knew a figure that used to be right.

This is the failure mode nobody can engineer away. A model's knowledge is fixed at training; tax law is not. Every budget, every finance act, every indexation order moves figures the model still believes. The Peru case is instructive twice over: the UIT is a multiplier, so one stale number silently corrupts every threshold, penalty and allowance derived from it.

It is also worth naming what this means for anyone comparing AI vendors. A model's tax accuracy decays continuously after its training cutoff, and the decay is invisible from the outside. A benchmark score published in January describes a different product by December.

Section 6What search does not fix: rules with conditions

This is the most important finding in the report, and the one that most directly answers what an accountant is actually for.

Seven questions defeated both arms. Searching the entire public internet fixed none of them. Here is every one:

Six of the seven are one error wearing different clothes: the model states the headline rate and drops the class of taxpayer that changes it. Filer or non-filer. Resident or non-resident. Commercial or private. Bank or other. The answer is not fabricated, it is not stale, and it will survive a citation check — the statute really does say 5%, for one kind of taxpayer.

The seventh case is different and worth its own note. Pakistan's securities rate is recorded in our own data as the bare string 0.15, with no unit. Both arms read the question and answered 12.5%. Our stored value means 15%, but a machine reading 0.15 has every reason to render it as 0.15%. That is our bug, not the model's: 159 facts across the corpus are bare numbers with no unit attached, and 3 are Excel date serials that were never converted. Small in number, but each one is a fact that will be read wrongly by the very audience we built the data for.

Section 7The confidence trap

We asked both arms to say how sure they were. The closed-book results are, frankly, better than we expected:

Closed-book, the model is well calibrated: when it says it is sure it is wrong 7% of the time, when it hedges it is wrong 18%, and when it says it is unsure it is wrong half the time. That is a genuinely useful signal, and it deserves saying plainly because the popular account of language models says the opposite.

Then search changes the picture in a way nobody should ignore. High-confidence answers went from 14 to 45. The error rate among them did not improve — it went from 7% to 9%. Having found a source, the model became sure; but the sources it found were often a professional summary that carries the same simplification the model already had. The count of confidently-wrong answers went from one to four.

So retrieval improves the average answer while degrading the usefulness of the model's own hedging. The user gets a better answer and a worse warning signal at the same time. Anyone building "AI checks its sources, so it must be right" into a workflow should sit with that.

One more number for the reliance question: closed-book, the model volunteered that a figure might have changed since training on 40% of answers, and flagged 48% as worth verifying. That is honest behaviour. It still answered 97% of the questions.

Section 8The source problem: the authorities are closed

We recorded what the searching arm actually cited, and this is the finding with the widest implications beyond us.

A primary source was reached for 15 of 60 questions — one in four. And the substitution is concentrated to a degree we did not anticipate: 26 of the 60 answers, 43% of the whole benchmark, cite the same website — PwC's Worldwide Tax Summaries.

It was not for want of trying. The run recorded being turned away, by name, from:

The practical consequence for a reader deciding what to trust: "the AI cited a source" and "the AI read the law" are not the same claim, and today they differ three times out of four.

Section 9Where this sits against everything else published

Our two central findings were arrived at independently, on our own data. Both turn out to be well supported by peer-reviewed work we found only afterwards, which is the outcome we wanted — a result nobody else has seen is usually a result that is wrong.

Finding: search fixes staleness and little else. FreshQA (Findings of ACL 2024) tested GPT-4 on the same day with and without search. On fast-changing facts it went from 26.0% to 67.7%, a gain of nearly 42 points. On never-changing facts it went from 92.7% to 96.0% — a gain of 3. Retrieval is transformative exactly where facts move and close to irrelevant where they do not. Our tax result is the same shape at a smaller magnitude.

The same paper records something that should end the "just make the model bigger" argument for tax: "there are flat scaling curves on questions that involve fast-changing knowledge: simply increasing the model size does not lead to reliable performance gains."

Finding: search does not stop confident error. Magesh et al. (Journal of Empirical Legal Studies, 2025) preregistered 202 queries against the commercial legal-research tools that are the closest existing analogue to what we are building:

Their conclusion, verbatim: "RAG systems are no panacea." Note the uncomfortable detail — two of the three retrieval-backed commercial products were less accurate overall than plain GPT-4, partly by refusing more often. Anyone quoting this study as "retrieval fixes hallucination" has not read the table.

The closest tax comparator, and why our number is not on its scale. TaxCalcBench (Column Tax) has models compute complete US tax returns, graded deterministically. Its live leaderboard shows the same retrieval effect we measured, in the same direction, on the same vendor's models:

Two further reference points for what closed-book models do on legal and factual recall. Dahl et al. (Journal of Legal Analysis, 2024, over 800,000 queries) found legal hallucination rates "between 58% of the time with ChatGPT 4 and 88% with Llama 2". And on LegalBench's sara_numeric task — given the statutes, state the tax owed — GPT-4 scored 8.3%, while on sara_entailment, deciding whether a rule applies at all, it scored 86.8%. That 78-point gap between reasoning about a rule and producing the right figure is the single most striking number in the literature, and it points the same way our Section 6 does.

One paper worth quoting to any accountant weighing this up. Cheng et al. (COLM 2024, Outstanding Paper) open with tax as their motivating case: "imagine a layperson using an LLM for tax advice, without realizing that the effective cutoff of the tax code is 2022 and thus outdated — despite the fact that the reported cutoff is advertised as 2023." A companion paper found LLaMA-2's knowledge peaks around 2019 against a stated 2022 cutoff. A model's advertised knowledge cutoff is not the date its tax knowledge actually stops.

And a warning about maintained data, which cuts against a lazy reading of our own conclusion. The HoH benchmark (ACL 2025) found that "even when current information is successfully retrieved, the mere presence of outdated information in the context leads to at least 20% performance drop" — outdated context is worse than no context. That is an argument for a corpus that is dated and maintained, not for one that is merely large. It is also a direct warning to us: our 66,253 facts include many nobody has looked at since they were written.

Section 10How our own drafts scored, judged by accountants

Separately from the benchmark, fourteen accountants reviewed 2,713 of our AI-drafted facts and recorded a verdict on each.

About three in four facts were right first time; one in four needed an accountant to change something. Two thirds of the changes were to the figure itself. A second and more generous reading — counting a citation-only fix as "the number was right" — gives 82.8%. Both are true, they answer different questions, and neither should be quoted without saying which.

The average hides the actual finding. Value accuracy by country ranged from 99.1% to 50.5%:

The spread is not reviewer strictness. It tracks how much of a country's tax law is published in a form a machine can read, which is the same finding Section 8 arrives at from the other direction.

Section 11What the AI actually gets wrong

Pooling the reviewers' 698 corrections with the benchmark failures gives a consistent taxonomy. In descending order of how often it happens and how hard it is to catch:

That last instrument gives a third data point worth stating, because it isolates the model from the retrieval problem entirely. Given a correct, accountant-reviewed guide directly in context, three models still deviated from it:

Same 31 questions, same guides, two runs each. Two cautions on reading it: at 62 runs per model these differences are inside the margin of error and should not be read as a model ranking, and the bigger model doing worse is interesting rather than established. The finding that is solid is the floor: even handed the correct answer in context, roughly one answer in six drifts from it. Supplying good data is necessary and not sufficient.

Section 12Which model wrote it? We can bound it, not name it

We expected to name the model in this report. We cannot name the version, and the honest account of why is more useful than a guess would have been.

No record exists. The drafting was done by agent sessions doing web research, not by a script calling an API with a logged model parameter, and the output files were never committed. We checked every layer that could have captured it: the audit log for all 620 authoring events records {slug, facts, topic, source, jurisdiction} and no model; no provenance column exists on the facts, the skills or the change sets; the signed ingest receipts record who verified, never who generated; the one model column that exists anywhere in the schema is an empty table.

And the working assumption was wrong. We believed the drafts were Claude Opus 4.8. The chronology rules that out for most of the corpus: the first mention of that model anywhere in our codebase is 9 June 2026, by which date 941 of 1,928 guides already existed. The step that produced the facts the accountants actually reviewed is pinned in code to a different model, claude-sonnet-4-6, and has never been changed. The only artifact in the entire company that names anything is one Korean review workbook referring to "Claude's proposed values" — which establishes the family and no version.

What we can do instead is bound it by dates. Setting the public release dates of each model against our own guide-creation dates rules out most of the possibilities:

The consequence is unambiguous. 901 of the 1,230 guides drafted before review — 73% — were written before Claude Opus 4.8 existed. The largest single batch, 350 guides on 13 April, predates even Opus 4.7 by three days; the newest model available that day was Opus 4.6. Only the 321 guides from 29 May onwards could have used 4.8 at all, and nothing records whether they did.

So the correct statement is: drafted between April and June 2026 by Claude-family agents doing live web research, on models available at the time — Opus 4.6 or Sonnet 4.6 for the bulk of the corpus; exact versions not recorded and not reconstructable. We have said this rather than the version we assumed, because a report about other people's unverified claims cannot rest on one of our own.

Model release dates above are from public secondary sources rather than a vendor changelog, and are stated to the day only where those sources agree.

One thing the same investigation did establish, and it is good news: the drafting was genuinely grounded. Of the surviving draft artifacts, 100% carried a legal reference and 72.7% carried a live source URL at the moment of drafting, and they cite documents that could not have come from a model's memory — including a District of Columbia payment booklet whose filename is date-stamped weeks before the draft was written.

Section 13The number that reframes all the others

Everything above concerns facts somebody checked. Here is the whole corpus:

The other 94.9% does not have an accuracy rate of 74%. It has an accuracy rate of unknown. Applying a reviewed subset's number to an unreviewed corpus is the single most tempting mistake available to us, and we are not going to make it in our own report. Kenya is the sharp example: fourteen published guides, 156 facts, zero reviews by anyone, ever.

This is why the benchmark in Sections 3 to 7 matters more than our own 74.3%. It measures the thing that scales.

Section 14What we got wrong about our own data

An earlier draft of this report contained a finding that the United Kingdom showed 182 facts confirmed, zero corrections and zero annotations, and drew the obvious conclusion about the reviewer.

That conclusion was false, and the fault was ours. The UK facts were imported with a flag that stamped every row "correct" without a human ever seeing them, from a workbook whose every row read "Pending". Our own records show the reviewer, James Power, returned two genuine corrections — the two the founder remembered independently. Our import overwrote his actual work with a fabricated hundred percent and then displayed it on a dashboard.

We are recording this in the report rather than quietly fixing it first, because a research report from a company whose product is provenance should show what happened when its own provenance failed. We came within one edit of publishing a named professional's supposed negligence that was in fact our software's.

Section 15Limits of this study

Stated plainly, because every one of them would be found by a sceptical reader anyway.

Section 16What is still open

Section 17What this says, plainly

A frontier model is a competent tax researcher and an unreliable tax authority. Asked a question about a rate or a threshold, it will answer nearly always, and on generous terms it will be right roughly three times in four from memory and closer to seven in eight if it can search. For a great many purposes that is genuinely useful, and it is better than the popular account of these systems suggests.

It is not what reliance requires. The residual errors are not random noise that more compute will sand away — they are structural, and they cluster in one place. The model is good at the rate and bad at the exception. Searching fixes what has gone out of date; it does not fix a rule that has a condition attached, because the summaries it can actually reach have already dropped the condition. And having searched, it becomes markedly more confident without becoming correspondingly more correct.

The gap that remains is therefore not a knowledge gap. It is the difference between recalling a rate and asking who the taxpayer is. That is the accountant's question, and nothing we measured closes it.

Two things follow for what we build. First, the binding constraint is not more guides; it is machine-reachable primary sources, because the world's tax authorities are closed to agents and the industry has quietly standardised on one Big Four summary as a substitute. Second, the review that matters is not "is this figure current" — search handles that — but "which taxpayer does this figure apply to". That is where a named professional adds something a model demonstrably does not have, and it is where our reviewers' corrections concentrated too.

There is a third implication we did not expect to find, and it is a caution rather than a strategy. A large corpus of unmaintained facts is not obviously better than no corpus: the published evidence is that stale material sitting in a model's context is worse than an empty context, because the model reconciles the two and loses. We have 66,253 facts and human eyes on 5.1% of them. The work that matters is not the next ten thousand facts. It is dates, sources and review on the ones we already have.

OpenAccountants Research Report OA-RR-2026-02 · 13 August 2026 · Figures verified against the production database on the date of publication. Reviewer names are published with the reviewers' work; corrections and disputes to the research team.