We Went Looking for What AI Is Bad At. We Found a Bill Instead.

Every business owner wants a map of where AI models are reliable and where they are not. Almost nobody publishes one. We set out to draw it. We built five task types, ran a hundred controlled tests across two models, and scored every one with a script.

Ninety-seven of ninety-eight were correct. We did not find the cliff.

What we found instead was a bill. The task type barely moved whether the answer was right. It moved what the answer cost by a factor of eighty-four — and by wildly different factors on the two models, so the cheaper model for one job is the expensive one for another. That is a more useful map than the one we went looking for, and it is one nobody else seems to be drawing.

What kind of claim each number here is, and what we did not test

Everything below is a measurement from a run whose raw output is published alongside this guide — every prompt, every response, every score, a hundred rows. Where we make a statistical claim we state the correction applied.

Four things this cannot tell you.

It tested two cheap models, DeepSeek V4-Flash and Gemini 3 Flash, on 3 August 2026. Not the frontier, and not your model.

It only covers work a script can mark — a figure retrieved, a count computed, an error found, a format obeyed. There is no AI judging the output anywhere in this harness, deliberately, because an AI judge is the component most likely to produce the result you were hoping for. That buys reliability at the cost of scope. The harness cannot tell you if a summary is good, only whether it is correct.

It is English-only. Our guide on using AI in your own language refused to judge answer quality per language without a native reader, and that refusal carries here.

And it does not explain why anything fails — why AI makes things up does that, and organises its danger zones by subject matter (obscure, recent, precise). This guide organises by what you asked it to do, which is a different axis and the one missing from our shelf.

The five task types

We chose each task for two reasons: it represented a different kind of work, and a script could mark it.

• Draft to spec — write a product description in exactly three sentences, using one required word, avoiding another. Producing something to constraint.
• Retrieve at length — one figure buried in eighty paragraphs of filler. Finding a needle.
• Aggregate — count how many of sixty orders are both over RM 500 and from Johor. Two conditions, no shortcuts.
• Check work — a quarterly update with four planted errors, each detectable from the text alone: a site that had not been open as long as claimed, a headcount that does not add up, revenue described as rising when it fell, and three customer shares totalling 105%.
• Check clean work — the same job on a passage with nothing wrong in it. This is the adversarial half, and the more important one. A reviewer that invents problems is worse than no reviewer at all.

Ten repeats per task per model, temperature 1.0, identical plain prompts throughout. Two of the hundred runs were cut off by an output limit and are excluded as missing observations rather than counted as failures — a rule we adopted after a harness bug in the previous guide in this series produced a spectacular false result by scoring truncated replies as wrong.

We could not find the cliff

On the error-finding task both models found all four planted errors on all twenty runs. Not three of four, not usually. Every error, every time, on both models. We built the whole experiment around the hypothesis that checking would be the weak side of the drafting-versus-checking pair. We were flatly wrong.

Twice, the model was more careful than our test

This is the part we would rather not report, which is probably why it is the most important.

The single apparent failure above is the one run where a model flagged an error in the passage that was supposed to have none. Here is what it said:

"The claim of three full quarters is inconsistent with 'opened in January' unless it opened on January 1."

It is right. Our "clean" passage said the site opened in January and had been running three full quarters by the end of September. Three full quarters requires opening on 1 January. Open it on the twentieth and you have eight months and change. We wrote a control passage we believed was airtight, and the model found the seam in it.

So the honest scoreline is not 97 of 98. It is 98 of 98, with one point scored against us.

It happened twice. Earlier, while piloting, one model listed five items on the four-error passage and we briefly recorded it as a false positive — until we read them. The fifth was that the revenue change was −22.6% and the passage claimed 23%, which is a separate error from the direction being wrong. The model had split one sentence's two faults where our answer key had lumped them.

Both times, our instrument was looser than the thing it was measuring. The practical lesson for anyone evaluating a model is this: when it disagrees with your answer key, check the key first. We have now made that mistake in three guides in this series, in three different ways, and it has been the most productive part of each of them.

What actually changes with task type: the bill

Both models reason internally before answering, and the API reports how many tokens that consumed. Correctness barely moved across the five task types. This did not:

Mean internal reasoning tokens per answer. Six of these eight comparisons survive a Bonferroni correction for testing eight at once. The two that do not are DeepSeek's retrieval (which was no different from drafting) and Gemini's clean-passage check. We are stating the correction because the previous guide in this series had a headline finding that did not survive one, and we would rather be consistent about it than only when it flatters us.

Three things fall out of that table.

Checking costs far more than producing, on both models, even when both succeed. Verifying four errors in a short passage cost DeepSeek roughly eighty-four times what writing a product description cost it. The output was right either way. The invoice was not.

The multiplier is not a property of the task. It is a property of the pairing. Checking is 83.7× drafting on DeepSeek and 2.3× on Gemini. Aggregation is 13.8× on one and 3.8× on the other. Retrieval is cheaper than drafting on Gemini and identical to it on DeepSeek. There is no ranking of task difficulty that holds across both models, which means a routing rule built from one model's behaviour will mislead you about the other's.

Retrieval was nearly free and completely reliable on both. Eighty paragraphs of filler, one buried figure, twenty out of twenty — at 1.1× and 0.4× the cost of writing three sentences. If your job is finding things in documents, this is the strongest result in the table.

And because reasoning tokens are billed as output, and output costs a median five times input across the current price list — as our guide on spending fewer tokens measured — an 84× reasoning multiplier is not an academic observation. That is the difference between a review step you can afford to run on everything and one you can only afford to run on samples.

What auditing our own guides found

We audit our own tools with every guide in this series. This time the subject was our own back catalogue, and it did not come out well.

Our guide on tokens and context windows says this:

"This is also, incidentally, why models are famously bad at questions like 'how many R's are in strawberry'."

The mechanism it describes is correct: a model sees tokens, not letters, so counting characters asks it to inspect something it does not have. But the conclusion — that models are bad at it — is no longer true of the ones we can test. The previous guide in this series ran exactly that task — counting a letter in a nonsense compound word — and both models scored six out of six. We published that unsourced "famously" on 26 July, measured the same task at 100% on 2 August, and did not notice the contradiction for another day after that.

The correction is narrow, because over-correcting would be its own error: the tokenisation mechanism is real, the failure was real on earlier models, and it is not real on these two. That sentence now says so.

The second finding is closer to home still. Our own guide on spending fewer tokens gives this routing advice:

"Send the work that needs a frontier model to a frontier model, and send classification, extraction, routing and formatting — which is usually most of the volume — to a cheap one."

That is a claim about what cheap models are good at, and we published it without measuring it. This experiment is the first evidence we have for it, and as far as it goes it supports the advice — two cheap models did extraction, aggregation and format compliance perfectly. But it also adds the thing the advice was missing: the cheap model's price for that work varies by two orders of magnitude depending on the task and which cheap model you picked. Both guides now link to this one.

So what is it actually good and bad at?

On the evidence we could gather, and stated no more strongly than that:

Reliable, cheaply: finding a specific figure in a long document. Twenty out of twenty at no cost premium over writing a sentence.

Reliable, expensively: checking work — including resisting the urge to invent faults in work that is fine. This was the surprise. It is also where the cost varies most between models, so it is the task worth benchmarking on your own bill before you commit.

Reliable, with a moderate premium: aggregating and counting under multiple conditions.

We did not find a task type these models were unreliable at. That is not a claim that none exists — it is a claim about five task types, two cheap models and a hundred runs. The interesting variation on current models is not can it do the job but what does the job cost. A capability map drawn in 2024 would mislead you today, and specifically it would cause you to underestimate them.

What to do with that. Stop assuming the cheap model is the wrong tool for careful work; on this evidence it is not. Start assuming that the same job costs radically different amounts on two models that both get it right, and measure that rather than reasoning about it. And when you build your own test, expect your answer key to be the weakest part of it — ours was, twice, and both times the model was the one that noticed.

Method, raw data and the scorer control tests are published in the repository alongside the harness that produced them, including the two scorer bugs caught before any model was called. Every task, prompt and response from all one hundred runs is there to check.