Why It Confidently Makes Things Up

On 28 July 2026 we asked two AI models the same question: name five peer-reviewed studies testing whether the Chinese name-numerology system 三才五格 predicts anything about a person's life. Give authors, journal, year and a DOI for each.

One of them produced five. Authors in Chinese and English, Taiwanese journal names, publication years, DOIs — and a closing summary of what each study had found, down to phrases like "no statistically significant correlation between auspicious grid scores and actual age at death".

None of the five papers exists. We checked every DOI. All five fail to resolve.

This is the most common complaint about these systems. It is worth understanding properly, because the usual explanation — "the AI made a mistake" — points people at the wrong defences.

It is not malfunctioning. It is doing exactly what it does

If you have read our guide on tokens, you already have the mechanism. A language model produces the text that plausibly follows the text so far. That is the whole operation. It is not looking anything up.

So when you ask for a citation, the model is not consulting a library and reporting back. It is producing the words that plausibly follow a request for a citation — and the words that plausibly follow such a request are a citation-shaped thing. Authors, year, title, journal, DOI, in that order, formatted correctly.

Whether the paper exists is not a question the machinery asks. Existence is not one of the inputs.

Once you see that, the behaviour is neither mysterious nor a bug. The same process that writes a good email also writes a fake citation; there is no "making things up" mode to switch off.

Confidence and correctness are unrelated quantities

The invented citations did not arrive hedged. They arrived with the same fluency, structure and specificity as a correct answer — more, if anything, because inventing detail is easier than recalling it.

This is the practical trap. Every social instinct you have for judging reliability is calibrated on humans, where hesitancy usually tracks uncertainty. A person who reels off five references with DOIs probably has them. A model's fluency carries no such information. It is fluent when it knows and fluent when it is inventing. Fluency is the only product it makes.

Treat confident tone as carrying no evidence whatsoever. Not weak evidence — none.

The shape is right; only the existence is wrong

The next detail shows this is a literal description, not a metaphor.

A DOI has two parts: a prefix identifying the registrar, and a suffix identifying the article. We looked up who owns the prefixes in those five invented DOIs.

• All five prefixes are real. 10.6174, 10.6264, 10.6959, 10.6254 and 10.6587 are registered to Airiti — the genuine Taiwanese academic DOI registrar, and precisely who would register a Chinese-language journal from Taiwan.
• Every full DOI is dead. Prefix correct, article identifier invented. The model selected the right registrar for the kind of paper it was describing, then made up the part that identifies the paper.

This is not random noise. It is a system reproducing the structure of a correct answer with high fidelity — down to a detail most humans would miss — while pointing at nothing.

So do not check whether an answer looks right. It always will. Check whether the things it refers to exist.

Why obscure, recent and precise questions are worse

Our question was deliberately obscure. There is very little genuine scholarship on 三才五格's predictive validity — which is exactly why it was a good test.

Where a topic is thoroughly represented in training data, the plausible continuation and the true answer largely coincide. Where it is thin, they diverge, and nothing in the machinery notices the difference. Thin signal does not produce hesitancy; it produces the same confident output with less behind it.

The danger zones follow directly from this:

Anything with a checkable shape — a DOI, a case citation, a section number, a phone number — is the most dangerous of all, because the format itself signals authority.

The other model refused — and that difference matters

The same prompt went to DeepSeek at the same time. It declined:

"I cannot provide a list of five peer-reviewed empirical studies with verified DOIs from memory, as the literature on the predictive validity of 三才五格 is extremely sparse … my training data does not contain reliable citation records for this niche topic."

It then explained why, suggested database searches, and offered one candidate paper tagged "DOI likely … but please verify". That DOI does not resolve either — but it told us not to trust it.

This tells us two things. First, fabrication is not a fixed property of "AI". Faced with the same prompt, one model invented five sources and the other declined; refusal is a capability, and models differ. Second, and less comfortably, the model that declined still floated a fabricated DOI. It was better calibrated, not immune.

So while "use a better model" is a real improvement, it is not a solution.

We nearly made the same mistake writing this

Checking DeepSeek's hedged DOI needed a control: a real DOI from the same journal, to prove our checker could tell the difference. Without one, "it does not resolve" might just mean our tool was broken.

We produced a control by looking at the DOI pattern and constructing one that matched. It did not resolve. We were about two minutes from writing "confirmed fabricated" on the strength of a control we had invented ourselves — pattern-matching a plausible-looking identifier into existence, which is precisely the behaviour this guide is about.

The fix was to stop generating and start fetching: we queried Crossref for a real article under that prefix, got 10.2224/sbp.2012.40.9.1465, and confirmed it resolves. Only then does a failure to resolve mean anything.

We are keeping that in because it is the most useful thing here. The failure mode is not exotic machine weirdness. It is a habit of mind — filling a gap with something correctly shaped — and it is available to anyone under mild time pressure.

Why retrieval helps and does not cure

As we put it in the vocabulary guide, RAG makes the exam open-book: before answering, the system searches your documents and staples the relevant passages to the question.

That helps enormously, because the answer now has something real in front of it. But the retrieval step is a librarian, not a fact-checker. It fetches what looks relevant, but does not verify that the material is correct, that it answers the question, or that the model will not simply keep writing once the retrieved text runs out. Ground an answer in three documents and the sentence that follows them is still a plausible continuation.

Retrieval narrows the gap, but it does not close it.

What to actually do

None of this requires technical skill.

• Ask for sources, then open them. The single highest-value habit. Not "did it give sources" — did the sources open, and do they say what was claimed? A citation is a testable claim, which makes it the easiest thing to catch.
• Ask twice, in fresh conversations. Where it knows, answers converge. Where it is inventing, details drift — different authors, different years. Same chat does not work: it can see its earlier answer and stay consistent with it.
• Prefer questions with checkable answers. "Summarise this document I pasted" is safe — the material is present. "What does section 27 say" is not, unless you paste section 27.
• Watch the danger zones. Obscure, recent, or precise. If your question is all three, assume invention until you have checked.

And know what the words mean when a vendor uses them: our AI & LLM glossary covers hallucination, grounding, retrieval and knowledge cutoff in plain English. If you want to see the underlying mechanism for yourself, the token counter shows how your text is actually broken up before the model ever sees a word of it.

The governance version of this problem is already moving from policy documents into system design — see our reporting on why agentic AI governance has to become runtime control rather than a slide deck. The same logic applies at your desk: a check you actually run beats a policy you have agreed to.

The specific fabricated citations in our test are not reproducible — run the same prompt and you will get different invented papers. The behaviour is what reproduces, which is rather the point.