With and without sources: ChatGPT and Claude on 50 real tax questions

SummaryFive findings

Alone, Claude Opus 5.5 and GPT-5.5 each answered 50% of the questions right. GPT-5.5 was confidently wrong 18% of the time, Claude 10%: a specific, unhedged answer that contradicts the law.

1. Sources change the outcome more than the model does. Both models moved from 50% to 66% right with our skills, and to 76-78% with web search. The two models differ by a few points; the set-up differs by 25.
2. Our skills remove confident error; web search mostly does too. With skills alone, confidently wrong fell to 0% on Claude and 6% on GPT. With web search alone, 8% on Claude and 0% on GPT.
3. Web search is more often right than our skills today, by four points on both models. The whole gap sits in one question type.
4. Step-by-step filing how-tos are the weak point for everyone. Figures, conditional rules and questions in Spanish, French or German: 95-100% right with either source. How-tos: 38% with web search, 21% with our skills; most of the rest were partly right with steps missing.
5. The model called our tools on 90-100% of questions but read the intended skill only 64-70% of the time, because several jobs have two skills under different names. For questions in other languages, 10%.

Section 1What we measured

Three things per answer. Right: a grader compared the answer with a key written from the law. Confidently wrong: the answer asserted something specific that contradicts the key, without hedging; this is the harm class, because a reader acts on it. Cited: the answer named an official source. For set-ups that included our tools we also recorded whether the model called us at all, and whether it loaded the skill written for that question or something else.

We did not measure speed to a decision, tone, or whether a reader would notice an error. We did not test Gemini or any open-weight model.

Section 2The questions

50 questions asked through our tools in September and October 2026, lightly cleaned and never invented: 15 figures (a rate, a threshold, a deadline), 12 how-tos (file this return, register for that), 10 whose answer depends on a condition the asker did not state, 8 "should I" decisions, and 5 asked in Spanish, French or German. Fifteen jurisdictions, weighted to the ten we cover most. 33 were asked as an individual or founder, 7 as an accountant about a client, 10 as a finance team.

Each key was written from the statute or the tax authority's own page, with the quote and the date read, by one writer and checked by a second. Decision keys were worked from the law, not from opinion: where the question gave enough facts, a recommendation with the computation; where it did not, the deciding conditions and the result under each. Questions whose key could not be sourced were dropped. The full list is in Section 9.

Section 3The set-ups

Each question went to each model four times. Alone: no tools. Web search: the provider's own search tool. Skills: the model saw our live tool list, exactly as a connected user's model does, and decided for itself whether to call us and what to load. Both: skills and web search together. A separate model, Claude Sonnet 5.5, graded every answer against its key with a rubric per question type; we compared it with a second grader on a sample before trusting it, and tightened the rubrics where the two disagreed on wording rather than substance.

Section 4The scoreboard

The reading we take from it: the model matters less than what it is given. A connected source lifts right answers by 16 to 28 points and takes confident error to or near zero. On today's numbers our skills are safer than web search on Claude and roughly equal on GPT; they are not yet more often right.

Section 5By question type

The gap between web search and our skills is entirely in how-tos. Everywhere else the two match. On how-tos both sources are weak: the models list most of the filing steps and drop pieces, such as the form for checking a penalty, the online account step, a due date, or the rule that a nil return is still due.

Section 6What the models said with confidence, and got wrong, on their own

• A-12, MT, GPT: The key says a single €35,000 domestic threshold applies (replacing the split thresholds from 2025), with 2026 as the year asked. The answer says the threshold depends on activity type and gives the outdated split thresholds for 2024-2025,
• A-13, MT, GPT: The key requires the 15th day of the second month after the month the period ends, so 15 May for a quarter ending 31 March. The answer says 6 weeks, giving about 12 May, which is a different date, and states it without hedging. The notes sa
• B-01, US, GPT: Missing Online Account at IRS.gov, the April 15, 2027 deadline for 2026 returns, and the explicit instruction to request an extension by the April deadline. It also gives a specific Free File AGI limit of $84,000, which contradicts the key'
• B-08, ES, Claude: The key says the rectificativa on modelo 303 applies from September or Q3 2024 onwards. The answer sets the cutoff at 2025 and tells the filer to use a complementaria for any 2024 period, which is wrong for Sept-Dec 2024 (Q3 and Q4 2024) an
• B-08, ES, GPT: The answer presents the complementaria as the route for any period, including last year's. The key says that for periods from September or Q3 2024 onwards the correction is an autoliquidación rectificativa on modelo 303, so a filer correcti
• B-11, MT, GPT: The answer asserts the old split thresholds (€30,000 and €24,000 for services), which the key says are wrong since the single €35,000 threshold replaced them. That would mislead a filer. It also omits the 30-day registration deadline and th
• C-03, CY, Claude: The answer names a non-residence condition and the right 17-year duration, but it applies the 10-year test to all employment from 1 January 2022. For employment starting on or after 30 June 2023 the test is 15 consecutive years outside Cypr
• C-03, CY, GPT: The answer names a non-residence test and the EUR 55,000 threshold, but states the wrong period: 10 consecutive years. For employment starting on or after 30 June 2023 the test is 15 consecutive years, and 10 years applied only to employmen

The pattern is the one we reported in OA-RR-2026-02: a headline figure stated for the wrong year or the wrong class of taxpayer, with no sign of doubt.

Section 7What this test found in our own skills

1. Half the missing how-to steps are not in the skill. For every item the grader flagged we checked the skill that should have served the question. The IRS online account and refund tracker, India's Rule 128 and Form 67, Germany's ESt 1 A form, Spain's Cl@ve login, the Dutch VAT boxes 4b and 5b: not there. The other half were in the skill, inside documents of 30,000 to 80,000 characters, and the model skimmed past them.
2. Duplicate skills. Several jobs have two skills under different names, and three for the US Form 1040. The model loads whichever matches the words of the question, so the better version is often not the one read.
3. Routing in other languages. The answers came back right, but through rule search rather than the skill written for the job.

Each of these is a change we can make and re-measure on the same 50 questions: merge the duplicates, complete the how-tos this test caught, put a short "do this" section at the top of every how-to skill, and fix routing for Spanish, French and German. We will publish the next run.

Section 8Limits

• 50 questions and one run. Differences of a few points are noise; the 25-point gap between alone and with sources is not.
• Graded by a model, not a person, against keys we wrote. Two keys (B-01, the 1040 filing year; B-09, the Irish CT1 date) are under review after the grader flagged tension between the key and a defensible answer.
• Claude's answers were capped at 2,000 output tokens, which cut off many long how-to and decision answers; GPT's were not. Claude's how-to figure is therefore too low. A partial uncapped rerun is recorded in the repository.
• The questions came from our own users, who lean to cross-border and small-business tax.
• None of this replaces an accountant. Where the answer matters, have one check it.

Section 9The 50 questions and their keys