We Tested the Prompting Folklore — 474 Answers Later, Nothing Had Changed

We told two AI models the same eight things, four hundred and eighty times, and then checked every answer with a script. Seven of those eight were prompting tricks people swear by: telling the model it is an expert, showing it a worked example, adding let's think step by step, saying please and thank you, offering it a $200 tip, telling it your job depends on the answer, and shouting IMPORTANT in capitals. The eighth was the plain instruction with nothing added.

474 answers were scored. All 474 were correct. Not one trick changed a single outcome, in either direction, on any task, on either model.

That is a duller headline than "these five prompts will 10× your output", and it is the one the data supports. The interesting part is not the null result, but how we got there. We spent most of this experiment failing to build a test these tricks could have passed, and along the way our own measuring equipment produced a spectacular false positive that we very nearly believed.

What kind of claim each number here is

A measurement is something this experiment produced, from a run whose raw output is published alongside this guide — every prompt, every response, every score, 480 rows.

A bound is what the sample size does and does not permit us to say. We state these rather than implying precision we do not have.

An open question is something we could not test, and there are several important ones.

Three things this experiment cannot tell you, stated before the results rather than after.

First, it says nothing about whether prompting tricks make writing better. Every task here was something a script could mark. A number is either right or wrong, a JSON object either parses or it does not, a sentence either obeys three constraints or breaks one. We deliberately used no AI to judge the output. An AI judge is the one component most likely to invent the result you were hoping for, a failure mode our guide on why AI makes things up takes apart. That decision buys reliability and costs scope. If a persona makes prose warmer or more useful, this experiment would not see it, and we make no claim either way.

Second, it tested two cheap models, not the frontier. DeepSeek V4-Flash and Gemini 3 Flash, on 2–3 August 2026, at temperature 1.0. Both, as it turned out, reason internally by default — which matters, and we come back to it.

Third, one piece of the folklore was already settled and is not re-run here. Our guide on using AI in your own language tested whether to write your instruction in English or in your own language, across two vendors and two tasks, and found no measurable difference. We cite that rather than repeat it.

Our own equipment lied to us first

This belongs near the top, not in a footnote, because it is the part most likely to be useful to someone running their own test.

The first pilot produced a striking result. It ran one attempt per condition, and on arithmetic Gemini got seven of the eight wrong. The only condition that produced a correct answer was emphasis, where the prompt began "IMPORTANT: FOLLOW THE INSTRUCTION EXACTLY". Shouting at the model appeared to be the one thing that worked.

That would have been a wonderful finding. It was entirely an artefact of our own harness.

Gemini 3 Flash is a thinking model, meaning it reasons internally before answering. The API's output-token limit has to cover both that internal thinking and the final, visible answer. We had set that limit to 700. Gemini spent 669 tokens thinking and had 27 left to answer with, so every response was cut off mid-sentence — before it reached the number. The scorer, correctly, found no correct answer in a truncated reply and marked it wrong.

The exception explains itself. Six of the seven failures ran to 24–28 output tokens and stopped mid-sentence, before reaching any number. The one prompt that told the model to follow the instruction exactly produced a seven-token answer — just 1075.20, no working — which fit inside the truncated budget. The trick had not improved accuracy. It had shortened the output enough to survive our bug.

Note how small the sample was, and how convincing it looked anyway: one run per condition, seven zeros and a one, and a tidy story to explain it. A single attempt per condition is not evidence, which is why the real run does ten.

We only caught it by reading the raw responses rather than trusting the scores. Two things followed. We set an explicit thinking budget so answers always have room. We also changed how cut-off responses are handled. They are now excluded as missing observations, not counted as wrong answers. A truncated reply is a fact about our harness, not about the model. Six of the final 480 runs were excluded on that rule, spread across four conditions and both vendors.

Raising the limit alone would not have saved us, which is worth knowing: Gemini's thinking expands to fill whatever it is given. At a 700-token cap it used 669. At 3,000 it used 2,877. Both still ran out.

We could not build a test the tricks could have passed

Before the real run, we needed a task these models get wrong often enough that a trick would have room to help. If the plain instruction already scores 100%, no trick can improve on it, and a null result would be a fact about our questions rather than about prompting.

We could not find one. Eight further task families were probed before the run — eleven variants in all, counting the three that made the final cut — both models, six runs each:

• Three-step arithmetic with a rate change and a percentage deduction — 100%
• Four-day arithmetic compounding a 15% price rise, a volume change, rent and corporation tax — 100%
• JSON with a required nested object and a field that must be a number, not a string — 100%
• A sentence under three simultaneous constraints; then under seven, including a banned letter — 100%
• Counting the letter r in a nonsense compound word, the classic tokenisation blind spot — 100%
• Reversing a word character by character — 100%
• An acrostic with per-word minimum lengths — 100%
• A margin trap: mark a price up 40%, then discount it 40%, which does not return you to where you started — 100%

The headline result, then, has a ceiling. And that ceiling is the real finding: on tasks a script can mark, these models were already acing the test before we added a single prompting trick.

Before believing any of that, we control-tested the scorers on inputs whose answers we already knew — eighteen cases, seven that should pass and eleven that should fail, run before every experiment. All eighteen behaved. A score of 100% and a broken detector look identical unless you check, a lesson we learned the hard way on an earlier guide when our detector was wrong before our finding was.

The result

Three tasks, eight conditions, two models, ten repeats each. 480 runs, 6 excluded as truncated, 474 scored, 474 correct.

The bound, stated properly. Sixty scored runs per condition with zero failures puts the true failure rate below roughly 5% for any single condition, and below 0.63% across all 474. And because the plain instruction already scored 100%, the largest improvement any trick could possibly have shown is zero percentage points. There was nothing left to win.

This does not mean the tricks are useless. It means that on this class of task, with these models, they simply had no room to work. Anyone quoting a percentage improvement for this kind of prompting should be asked what their baseline was.

The second finding, and why we are not publishing it as one

Correctness was flat, but we also recorded how many reasoning tokens each model burned before answering. There the numbers moved, and moved in ways that would make excellent headlines.

Offering DeepSeek a $200 tip made it think 63% harder. Threatening it with the sack: 51% harder. In the other direction, telling Gemini to think step by step made it think 33% less, and shouting IMPORTANT at it, 25% less — the opposite of what both tricks are supposed to do.

Those are real numbers from the run. They are also, on this evidence, noise.

The spread within each condition is enormous — for the tip condition the standard deviation, 425 tokens, is nearly as large as the mean, 490. Once you account for testing fourteen comparisons at once rather than one, none of the fourteen survives. Three would have cleared an uncorrected significance test — and they are exactly the three eye-catching ones above. That is precisely the mechanism by which a striking result gets published and then fails to replicate.

So we are reporting the numbers, reporting that they do not hold up, and drawing no conclusion from them. If offering a tip really does make a model think harder, this experiment did not establish it. It establishes that we nearly had a second headline before the statistics took it away.

Why "think step by step" has less to do than it used to

A structural observation might explain this, though it is not a formal result of the experiment. Both models we tested already reason internally by default. DeepSeek reports its reasoning tokens; Gemini spent 850 of them per answer in the plain-instruction condition without being asked to. The instruction that made chain-of-thought famous was aimed at models that would otherwise answer immediately. Told to think step by step, a model that was already thinking has been told to do what it was doing.

That is consistent with what we measured — no accuracy change, and if anything slightly less internal reasoning — but our data cannot prove the mechanism, and we are not going to claim it does.

What auditing our own tools found

We publish more than forty prompt-building tools, so before criticising anyone else's claims we read our own. One of them is the worst offender in this guide.

Our AI prompt builder told readers this, in an FAQ and again in its explainer:

"Internal benchmarks across multiple studies (2023–2025) show framework-structured prompts produce 30–60% more accurate, more relevant, and more usable output on tasks that have a clear 'done' standard."

"Internal benchmarks" means ours. We had never run one. There is no such study, no such dataset and no such measurement anywhere in this company — the experiment described in this guide is the first internal benchmark of prompting we have ever conducted, and it found a baseline of 100%, where a 30–60% improvement is not merely unproven but arithmetically impossible.

The same tool asserted that "roughly 70% of the time the second iteration is materially better than the first", that six frameworks "cover roughly 95% of practical use cases", and carried a heading promising frameworks that "reliably produce better AI output". None of those numbers had a source either. A neighbouring tool, our prompt framework builder, sold a "proven framework" and stated that giving the model a role "reliably sharpens tone and depth".

All of it is now corrected. The new text does not offer a softer number; it tells the truth. A structured prompt is a checklist to stop you omitting things. Its benefit is that your request is complete and easy to revise, and we do not have a figure for how much it improves output because nobody we can cite has produced one. Where our own experiment applies, it is linked.

We are not enjoying this. A guide about unfounded prompting claims that did not check its own tools first would have deserved everything it got.

So what should you actually do?

This is not an argument for writing worse prompts. It is an argument for knowing which parts are doing the work.

Be specific and complete. Every task in this experiment was unambiguous — the constraints were stated, the output shape was stated, and the models met them every time. That part reliably matters. It is not folklore, just the simple practice of saying what you want. If a checklist helps you avoid leaving something out, our structured prompt builder and system prompt builder are exactly that and nothing more — which is now what they say about themselves.

Do not pay for theatre. Nothing in our data suggests a tip, a threat or a please changes an answer. Two of them were associated with more internal reasoning, and while that did not survive our own significance test, more reasoning is never free — output tokens cost a median five times input tokens across the current price list, as our guide on spending fewer tokens measured. A ritual that might cost you money and demonstrably does not buy accuracy is a bad trade at any effect size.

Test on your own work, and check your instrument first. Our result covers three narrow, mechanically-checkable tasks on two cheap models. Yours may be harder, subtler, or judged by a human — all cases where these findings do not reach. If you do run your own comparison, control-test whatever is doing the scoring before you believe a flat result, and read a few raw outputs with your own eyes. Our harness produced an eightfold improvement that did not exist, and no amount of staring at the summary table would have revealed it.

And ask for the baseline. When someone tells you a technique makes a model 40% better, the question that settles it is what the model scored without it. If the answer is 100%, as it was here, the claim cannot be true.

Method, raw data and scorer control tests: the full 480-row output of this experiment is published in the repository alongside the harness that produced it, including the failed pilots. Anyone is welcome to check our arithmetic, and given what we found in our own tools this week, we would rather you did.