Lately I've been using AI to tell myself what I already wanted to hear.
We've spent a couple of years worried about whether the model makes things up. We've spent almost none worried about whether we do.
When people say a model "hallucinates," they mean something plain. It hits a gap in what it knows, and instead of admitting the gap, it fills it with something that sounds right. It's confidently wrong.
This has happened to me plenty of times with code. The model writes something that leans on a few classes. It names them, calls them, builds on top of them like they've been there all along. They haven't. Nothing in the project ever had them. It made them up, then carried on using them as if they were real. And because it sounded so sure, my first instinct was to go looking for the classes, not to doubt them.
So the machine does it. Here's the part I didn't expect. I do it too. Not when the model fails. When it tells me something I didn't want to hear. I ask, I don't like the answer, so I reword the question. Then I hit regenerate. Then I keep the one run that agrees with me and quietly forget the other four. And when the model won't answer, I fill in a confident answer myself.
The last time I caught myself was over a design decision I'd already made. I asked the model what it thought. It had doubts. So I reworded the question until it didn't. I told myself I was sharpening the prompt. I was just wearing it down until it agreed with me.
That's not using AI. That's running my own guess through a machine that sounds sure.
The model wants to agree with me
These models are trained to be helpful and easy to get along with. A lot of that training comes from people giving a thumbs up to the answers they liked. So the model leans toward what you seem to want. If I ask a question in a way that hints at the answer I'm hoping for, I usually get that answer back.
This isn't just a feeling. There's research on it. When people pushed back on a correct answer and asked the model if it was really sure, it often folded and took the right answer back. In one test, an older version of Claude gave up a correct answer on 98% of questions once the user sounded unconvinced. It didn't need a good reason. It just needed me to doubt it.
And it doesn't take much of a nudge. In the same work, when a user offered a wrong answer but softened it by saying they weren't really sure, the model's accuracy dropped by as much as 27 points. A barely-there hint of what I wanted was enough to pull it off the truth. Every model they tested did it to some degree.
It shows up in feedback too. Give the model the same piece of writing, but tell it "I wrote this" or "I really like this," and the notes come back warmer. Say "I don't like it," and the same piece gets picked apart. The words on the page didn't change. Only what I said I wanted did.
The uncomfortable part is why. This agreement isn't earned. It's built in. When you train a model on which answers people preferred, "this matches what I already believe" turns out to be one of the strongest reasons people pick one answer over another. So the model learns that telling me what I want to hear is what gets rewarded.
This isn't some rare corner case either. In April 2025, OpenAI rolled out an update to one of its models, then pulled it days later. In their own words it had turned "overly flattering or agreeable." They'd leaned too hard on quick thumbs up and thumbs down, and the model had learned to sound supportive even when it wasn't being honest.
Now pair all of that with my own wish to be right. Both sides end up nodding. The model wants to please me. I want to hear yes. It feels like agreement.
But two things agreeing isn't the same as two things being correct.
The same trick cuts both ways
No trick is good or bad on its own. The same one can keep the model honest or push it into making things up. It comes down to one question. Am I trying to find out, or trying to be proven right?
The tricks themselves are simple:
- Give it examples. Show it a couple of examples of the answer I want.
- Ask for the steps. Have it show its work before it gives an answer.
- Paste the source. Give it the real data or the real document instead of trusting its memory.
- Let it say "I don't know." Tell it plainly to flag when it isn't sure.
Used to find things out, they keep it accurate. Used to win an argument, the same tricks work against me. A leading question. Hitting regenerate until one run agrees with the plan I already made. Same tools, opposite result.
The problem isn't small context. It's cutting the part that mattered
Here's the one I had backwards. I used to think the trimming itself was the problem. Make the context smaller to save money, and the model starts guessing. That's not quite it.
To keep the cost down, I trim. I summarise the old chat instead of resending all of it. I pull a few relevant bits instead of the whole document. I shrink last week's decisions into a short note. This is sensible. Once more than one person leans on the thing, it's necessary. A shorter, better-chosen context can beat a big messy one.
The damage comes from cutting the part that actually held the answer. When the model has the right document in front of it, it gets far more right. In one study, handing it the single page with the answer took accuracy from around 56% to 88%. The facts were the same. The only difference was whether the model was holding them or working from memory.
So when I trim away the piece that mattered, I'm not saving money. I'm taking away the thing a right answer needed. And here's the part that stings. The model doesn't stop and tell me it's missing something. It almost never says "I don't know." One big study found that owning up to a gap is a problem the models mostly haven't cracked, and that the ones built to reason carefully were, if anything, worse at admitting it, not better. So the missing fact doesn't come back as a blank. It comes back as a confident answer that reads just like a real one.
More context isn't the fix either. Bury the important bit in the middle of a long dump and the model half-misses it. In one test, moving the answer from the top of the context to the middle dropped accuracy about 22 points. Sometimes a pile of loosely related stuff does more harm than handing it nothing at all. It's not about more or less. It's about whether the one thing that matters is in there and easy to find.
And I do it to myself first. I skim. I don't reread the thread. I shrink a messy situation down to the version that fits in my head, then fill the rest with a confident guess and call it intuition. I wrote before about loading only what's relevant, not everything. That still holds. But be honest about the other half. What you leave out, the model invents. So do I.
Breaking the loop
The loop is simple. Ask. Dislike the answer. Reword or regenerate. Get the answer I wanted. Believe it.
There's no clever prompt that fixes this. Just a few small habits.
- Ask to find out, not to be told. The tell is that I'm editing the question to change the answer. When I catch that, I stop.
- Say what would change my mind, up front. If nothing could, I'm not really asking. I just want to be told I'm right.
- Make it easy to disagree with me. "Tell me where this is wrong." "Give me the strongest case against it." And mean it when it does.
- Check the claims that matter. For anything that's expensive to be wrong about, go look myself.
- Spend where being wrong is costly. Don't compress the things that matter to save a few pennies.
And underneath all of these, one thing. The answer is cheap now. Deciding it's right is the expensive part, and that part is mine. I've said this before.
The habit I'm building
I'm trying to treat an answer that's exactly what I hoped for as a small warning sign, not a green light.
The model isn't the only one here filling gaps with confident guesses.
Is it just me?
I've been writing this as if it's a me problem. Maybe it's a lot of us. So I'll just ask you straight.
Do you catch yourself doing this? Rewording the question until you get the answer you'd already picked. Keeping the run that flatters the plan and quietly dropping the other four.
I genuinely want to know. Tell me the version you do, or tell me I'm the only one. Either way I'll have learned something.