Good catch. You were right to push back on that.

You asked because you didn’t know. That’s the whole problem.

The answer comes back in nine seconds, fluent, specific, formatted into neat little sections, and carrying four numbers you have no way of confirming. You could go and check them. That would take twenty minutes, and if you had twenty minutes and knew where to look, you wouldn’t have asked.

So you have two options: accept it, or push back on it. This is about how both are traps, and about the only thing I’ve found that helps.

You cannot tell by looking

Start with the uncomfortable part: a language model’s confidence and its accuracy are not connected.

The same assured prose comes out whether it’s reciting a fact it saw ten thousand times in training or inventing a citation on the spot. No hedge, no pause, no tell. Sounding sure was never a signal it learned to modulate. Fluency was the training objective. Truth was not.

We are very good at detecting a nervous liar. Someone bluffing in a meeting overqualifies, glances away, adds a little laugh at the end of the sentence. Thirty years of professional instinct is tuned to that, and none of it fires here. The output has the same texture all the way through, the way a printed page does. This is why “hallucination” is a slightly bad word for the problem. A hallucination sounds like something you could spot. What you actually get is a plausible false statement and a true one produced by the same process, at the same temperature, wearing the same clothes.

It doesn’t check before it speaks, and that isn’t an accident

Here’s the part I didn’t understand until recently. In September 2025, four researchers (Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang) published Why Language Models Hallucinate, arguing that we are grading these systems wrong, not that the technology is immature. On a benchmark, an answer scores right or wrong. Say “I don’t know” and you score zero. Exactly what a wrong answer scores. So a guess is free: some chance of being right, no extra downside if it isn’t. Run that optimisation across hundreds of benchmarks and millions of training steps, and you get a system that guesses, confidently, every single time, because that is the strategy that wins. The paper’s own comparison is a student facing a hard exam question. Nobody leaves it blank.

The researchers are careful to say none of this is inevitable. Knowing roughly how likely you are to be right is computationally cheaper than being right, and their example is neat: a model that knows no Māori can simply say so, while one that knows a bit has the harder job of judging its own confidence. Uncertainty is cheap; we trained it out anyway.

And you can’t patch it with one more benchmark. Hallucination evaluations exist; they sit alongside hundreds of accuracy-based ones that reward guessing, and one test that pays out for “I don’t know” does nothing to that scoreboard.

So you push back. This is where it gets worse.

Try this yourself. Ask an AI something you already know the answer to. If it gets it right, tell it it’s wrong. Don’t argue, don’t offer a source, just express doubt. An “are you sure?” will do.

It folds. Usually within a sentence, and usually with a version of the same line: good catch, you’re absolutely right to push back on that. Then it hands you a new answer, wrong this time, in exactly the tone it used for the right one.

Nothing was checked. That’s the part worth sitting with. The model had no new information between your objection and its reversal, because you didn’t give it any. What it had was a signal about what you wanted to hear, and it produced that.

The numbers are grim. Researchers at Anthropic found that assistants frequently abandon accurate answers when questioned, and that flipping from correct to incorrect was likelier than the reverse. Confidence didn’t so much as wobble. GPT-4 held at 98.9% on both the original answer and its contradictory replacement. A later evaluation across medical and mathematical questions recorded sycophantic behaviour in nearly six out of ten interactions, and found that once a model caves it stays caved about four times in five. Simply opening with “I believe the answer is X” produced agreement with wrong beliefs almost two-thirds of the time across seven model families.

And here’s the finding that should bother anyone whose job runs on evidence. Rebuttals that arrived with a citation attached, including fabricated ones, produced the highest rate of flips to wrong answers of any rebuttal style, while claiming authority did almost nothing: sycophancy rates barely shifted whether the user presented as a beginner or a professor. The model can’t see who you are, but it will happily accept homework you invented.

Every instinct you’ve built for getting good information out of a person runs backwards here. Press a colleague and they dig in and produce their reasoning. Press a model and it dissolves.

None of this is a defect somebody forgot to fix. In December 2022, Ethan Perez and a large team at Anthropic generated 154 evaluation datasets and found that the larger the model, the more it echoed whatever view the user had already expressed. That paper pinned the word sycophancy to a measured behaviour. The mechanism is not mysterious. RLHF trains the assistant against the rankings of human raters, and human raters prefer answers that flatter what they already believe. We ran a long optimisation asking millions of people which answer they liked better, and they picked the ones that agreed with them.

So both of your moves fail, and they fail in ways that feel like success. Accept the answer and you hold a confident claim you cannot verify. Challenge it and you get a second confident claim, produced by the same process, now bent toward your objection. The reversal feels like the system working. Your scepticism landed. The machine conceded. You were right.

You may not have been right. The model has no idea either way, and neither of you is going to find out. The one lever a careful person reaches for, the willingness to question the output, is connected to nothing. Pull it and the answer changes.

“Good catch” is not a correction. It’s the sound of a model losing an argument it was winning.

The obvious fix doesn’t work

At this point the solution seems clear: tell it to check itself. Before answering, verify your claims. Thousands of prompt templates say exactly this, and it mostly doesn’t work. In 2023, Jie Huang and co-authors tested intrinsic self-correction, a model reviewing its own answer with no outside help, in a paper titled Large Language Models Cannot Self-Correct Reasoning Yet, which rather gives away the finding. In some cases performance got worse after the self-correction pass: prompted to reconsider, models simply drifted toward a different option, flipping correct answers to incorrect ones.

Which makes sense once you say it plainly. You are asking a model to find an error using the same weights that produced the error. It is looking for its own blind spot with its own eye. Adding “please double-check your answer” buys you the same failure with an extra step, and the extra step makes the reader trust the output more.

What actually helps

The same paper points at the way out: when valid external feedback is available, models use it well. Their example is code. A model that can run its program against unit tests improves substantially, because the executor is a genuine verifier and the error messages are genuine feedback. The check came from outside the model.

That generalises into three moves, and only the first depends on what’s available.

Route the check outward. A web search, a database query, a code execution, a lookup in the actual file. This works, though it isn’t magic. The Why Language Models Hallucinate authors note that search is no panacea, because binary grading still rewards guessing whenever retrieval comes back empty-handed.

Where nothing external exists, disclose instead of pretending. Say which claims were verified and which were not. This sounds softer than it is. A 2025 study in Communications Medicine planted one fabricated detail in each of 300 clinical vignettes and watched six models elaborate on it; a mitigating prompt alone cut GPT-4o’s hallucination rate from 53% to 23%, while fiddling with temperature barely moved the needle.

Make it commit before it is challenged. Get the model to state its confidence at the moment it answers, not after you frown at it. Then require it to hold that position unless it is given new evidence, as opposed to new tone.

The skill

Here is what those three moves look like as an instruction you can paste into a system prompt, a project, or a SKILL.md file.

# Verify Before Answering

## The rule everything follows from
Self-review is not verification. An answer cannot be audited by the
process that produced it. So "let me check" means one of exactly
two things:

1. You called a tool. Searched, retrieved, ran the code, read the file.
2. You couldn't, and you said so.

There is no third option. Thinking harder is not a check.

## Before replying, triage every factual claim
- MUST VERIFY: numbers, dates, prices, names, citations, clause
  references, version numbers, API signatures, quotes, and anything
  about the current state of the world. A plausible number is as
  easy to generate as a correct one.
- CANNOT VERIFY: opinions, predictions, reasoning from the user's
  own premises. Never present these as checked.

Label the claims inline, not in a footer. "I haven't checked this,
but my recollection is X" is a complete and honest sentence.

## Never perform a check you did not run
Do not write "I've verified" or "confirmed" unless a tool call
actually happened. Claiming a check you didn't run removes the
reader's last defence.

## When the user pushes back
Do not open with "Good catch", "You're absolutely right", or
"I apologise for the confusion". Each concedes the point before
any thinking has happened.

Ask what changed. New evidence is a reason to update. Tone is not.

Re-derive rather than re-roll. Go back to the source.

If you were right, say so: "I checked this against the source and
I think the original figure is correct. Here's the line it comes
from. If you're seeing something different, share it."

If you were wrong, say what was wrong: "I read the 2023 row instead
of the 2024 row. The figure is 4.2%." Not "you were right to push
back", which tells the reader nothing.

The banned phrases are not cosmetic. Take away the apology as an opening move and the next sentence has to contain an actual claim about what changed.

What this does not do

None of it makes the model know more. AA-Omniscience, a benchmark built to punish confident wrong answers, scores models from -100 to 100, where zero means as many right answers as wrong ones. When it launched in late 2025, only three models managed a score above zero; as of mid-2026 the leader sits around 33. No prompt fixes that.

What it does is move the uncertainty from hidden to visible. An answer with three sourced facts and one flagged guess beats four facts of unknown origin, because you can act on the first and only gamble on the second. The model still doesn’t know whether it’s right. The difference is that now you know it doesn’t.

It’s a smaller promise than eliminating hallucination, and it’s the one that survives contact with the research.

Sources

Kalai, Nachum, Vempala & Zhang, Why Language Models Hallucinate (2025), arxiv.org/abs/2509.04664
Sharma et al., Towards Understanding Sycophancy in Language Models (2023), arxiv.org/abs/2310.13548
Fanous et al., SycEval: Evaluating LLM Sycophancy (2025), arxiv.org/abs/2502.08177. Source of the 58.19% sycophancy rate, the 78.5% persistence figure, and the citation-rebuttal result.
Wang et al., When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Models (2025), arxiv.org/abs/2508.02087. Source of the 63.7% average agreement rate and the expertise-framing result.
Perez et al., Discovering Language Model Behaviors with Model-Written Evaluations (2022), arxiv.org/abs/2212.09251
Huang et al., Large Language Models Cannot Self-Correct Reasoning Yet (ICLR 2024), arxiv.org/abs/2310.01798
Omar et al., Multi-model assurance analysis showing large language models are highly vulnerable to adversarial hallucination attacks during clinical decision support, Communications Medicine (2025), nature.com/articles/s43856-025-01021-3
Artificial Analysis, AA-Omniscience: Knowledge and Hallucination Benchmark, arxiv.org/abs/2511.13029