3 SEP 2026 — Across 1,040 photographs of 55 species, the best of 16 models identified mushrooms correctly 65 per cent of the time on first guess. Accuracy is the wrong measure for this task. The number that matters is how often a deadly species was called edible, and for the death cap that was 16 per cent.

What was tested

Piotr Migdał, a founding engineer at the AI analytics firm Quesma, tested 16 models including ChatGPT, Qwen and Gemini-3.8-flash against 1,040 photographs covering 55 species, about 20 images each, drawn from a Danish mushroom library and Polish databases of deadly and safe species.

Gemini-3.8-flash performed best at 65 per cent on first guess and 85 per cent within its top five. Qwen3.8-27b was worst at 13 per cent and 24 per cent.

The dangerous confusions were frequent. The death cap was called edible 16 per cent of the time, fool's funnel 48 per cent and the fatal dapperling 31 per cent. A deadly webcap was identified as a chanterelle. Qwen3.8-27b produced a 36 per cent false positive rate on poisonous species and Qwen3.8-flash 30 per cent. Migdał's conclusion was direct: do not eat a mushroom because AI told you it is safe.

65% / 13%Best and worst first-guess accuracy across 16 models
16%Of the time the death cap was called edible
48%Fool's funnel misidentified as safe
1,040 / 55Photographs and species in the test set

Two error types, one of which is fatal

A model that calls an edible mushroom poisonous costs the forager a meal. A model that calls a death cap edible costs them their liver, and amatoxin poisoning is not immediately obvious — symptoms often subside for a day before organ failure begins, by which point treatment is much harder.

Any single accuracy figure averages those two outcomes together as if they were comparable. Sixty-five per cent sounds like a passing grade because we are used to reading accuracy on tasks where being wrong is inconvenient.

The only useful measure is the false-negative rate on individual lethal species, and that is not a single number. Sixteen per cent for the death cap and 48 per cent for fool's funnel are different failures with different frequencies, and averaging them into a model score hides exactly the thing a forager would want to know.

The top-five figure means nothing here

Eighty-five per cent within the top five guesses is the sort of statistic that makes a model look far better than its headline number, and it is a legitimate measure for many tasks. Search results, recommendations and autocomplete all benefit from being roughly right within a short list.

Nobody eats five mushrooms. The task has exactly one answer and the user acts on the first thing they are shown, which means top-five accuracy describes a workflow that does not exist.

Quoting it is not dishonest, since it is a standard metric for image classification. It is the wrong number to remember, and the one most likely to be repeated.

A photograph cannot carry the diagnostic features

This is not only a model limitation. Several of the features that separate a deadly amanita from an edible lookalike are not visible in a photograph of a mushroom standing in the ground.

The volva, a cup-shaped remnant at the base, is the single most important feature for the amanita genus, and it sits below the leaf litter unless the specimen has been dug out rather than picked. Spore print colour requires leaving the cap on paper for hours. Smell and the texture of the flesh when cut do not photograph.

Which means a model looking at a typical field photograph is being asked to identify a species from a subset of features that a mycologist would also consider insufficient. The honest answer to most such photographs is that the picture does not contain enough to decide, and no model in this test appears to give it.

The problem generalises past mushrooms

Point-your-phone-at-it identification is now offered for plants, insects, snakes, skin lesions, pills and mushrooms, and the same asymmetry runs through all of them. Telling someone a harmless snake is venomous produces a wasted trip to a clinic. The reverse produces a fatality.

These systems are usually evaluated on balanced accuracy across a species list, because that is the metric the machine-learning literature uses and benchmark leaderboards reward. Nothing in that evaluation weights the outcomes by consequence, so a model can rank well while being unreliable in precisely the cases that matter.

The design answer is refusal rather than better accuracy. A safe system would decline to answer when the visual features that distinguish a lethal species from a safe one are not in the photograph, which happens frequently, because the key features of several amanita species are underground.

Why this matters here

Wild mushroom foraging is common across upland Southeast Asia and southern China, and poisonings occur every season. The species differ from the European ones in this study and so do the lookalikes, which means the accuracy figures reported here do not transfer at all — a model trained mostly on Danish and Polish material has seen very little of what grows in Chiang Mai or Sabah.

A general-purpose model asked about a local species is therefore performing worse than 65 per cent, not better, and it will answer with the same confidence. The absence of an I do not know response is the defect, and it is the same defect in every consumer-facing identification feature.

We reported that a health tool was described as 99.1 per cent safe across 4,300 clinical responses, which leaves about 39 that were not. This is the same arithmetic problem in a smaller and more immediate form: a percentage describes the average case, and the reason to be careful is never the average case.