Your AI doesn't have a hallucination problem

You ask for a piece about hiring your first marketer. What comes back has a paragraph about your pricing philosophy in the middle of it — fluent, confident, in your voice, and completely beside the point.

Everyone calls this hallucination. Everyone then reaches for the same fix: prompt harder. Add "stay on topic." Add "do not discuss unrelated subjects." Add it in capitals. Add it twice.

We did the audit. The model wasn't hallucinating. It wasn't even being creative. It was doing exactly what it was told, with exactly the material it was handed — and the material was wrong before the prompt was ever assembled.

TL;DR: A model writing off-topic is usually a prompt composition defect, not a generation defect. We traced every channel that put text in front of ours and found eight. One had a relevance floor. Seven did not. Fixing the composition — not the wording — took subject fidelity from 50% to 100% on our own regression fixtures, and the prompts for a piece with no extra context came out byte-identical to before.

Hallucination is the wrong word for this

A hallucination is an invention: a statistic that doesn't exist, a customer who was never a customer, a quote nobody said. That's a real failure mode, it's dangerous, and it needs its own defences — we run a separate grounding check for exactly that.

What I'm describing is different and much more common. Every sentence in that stray pricing paragraph was true. I had written it. I believed it. It was retrieved from my own knowledge, accurately, with the receipts attached.

It just had nothing to do with hiring.

Calling that a hallucination sends you looking in the wrong place. You start auditing the model's honesty when you should be auditing your own plumbing. The question is not "why did it make that up." The question is: who put that in front of it?

We went and counted

So we did the unglamorous thing. We took one piece of writing through the whole pipeline and logged every single channel that contributed text to a prompt. Not the ones we remembered — all of them, found by reading the code path rather than by asking ourselves what we thought was there.

There were eight.

Some were obvious: the knowledge retrieval, the style fingerprint. Some were not: a digest assembled two stages upstream and passed down, a set of records fetched for the outline planner and quietly reused by every section, an "interesting related material" pass nobody could remember approving.

Exactly one of those eight applied a relevance floor before handing its material over.

The other seven ranked. And ranking is the trap, because ranking always returns something. Ask a ranked retrieval for your five most relevant records on a subject you have never written about, and it does not say "nothing here." It hands you your five least irrelevant records with a straight face. On a subject you know well, that's fine — the top five are genuinely about it. On a new subject, the top five are whatever your brain happens to contain, and the model is now looking at your pricing argument while trying to write about hiring.

Nobody wrote a bug. Every one of those seven channels worked as designed. The defect was in the composition.

A new subject has to be allowed to return nothing

This turned out to be the load-bearing fix, and it is genuinely uncomfortable to ship.

An empty retrieval feels like a failure. Every instinct in a product team says don't show the user nothing. So the natural design is a nearest-neighbour search with no floor, which structurally cannot return nothing — and that instinct is precisely what was poisoning the prompt.

So we put a floor on every channel. Now, when you write about something genuinely new to you, the brain returns nothing and says so: I don't hold anything on this yet. The draft gets written from what you tell it in the moment instead of from a bin of near-misses.

The thing I didn't expect: writers like this. "I have nothing on this" is useful information. It means you're breaking new ground, which is usually why you sat down. The old behaviour — quietly padding your new piece with your old opinions — was the thing making drafts feel generic, and no one could name why.

The piece isn't about what you typed

Here's the second finding, and it's the one I'm most embarrassed about.

We stored the writer's first sentence as the piece's subject. Reasonable, until you look at what people actually type. When you turn an article into a LinkedIn post, the first thing you type is something like "Write a LinkedIn post from this article."

That is an instruction. It is not a subject. But four different parts of our pipeline had adopted it as one — so they went off and retrieved everything we knew about writing LinkedIn posts, and none of it was about the article you had attached.

We checked this across a sample of eight real records. Read cold, zero of the eight gave the right subject. Resolved properly — following the attachment to what the source piece was actually about — eight out of eight did.

The fix isn't clever. It's a lookup that follows the reference instead of trusting the sentence. But you only get to write it once you've stopped calling the symptom a hallucination.

Precedence, as structure rather than as prose

The third piece was about what happens when two sources disagree.

If you attach a report and also have a brain full of opinions, which wins? The old answer was: we mention both and hope the model works it out. Which is not an answer, it's a shrug — you have handed the model an ambiguity and called the result its fault.

The new answer is a ledger with four tiers in a fixed order: what you supplied outranks what you stated outranks the subject's own material outranks your style. It's not phrased as a request in the prompt. It's structural: when you supply source material, the brain's own contribution is capped — because you have handed over something specific and the whole point is to work from it.

One composer builds the block, every generation call renders the same one. Which fixed a bug we'd been living with for months: the page you attached got read once by the outline planner and then evaporated. Every section prompt after that was written by a model that could no longer see the thing it was supposed to be summarising.

Checking afterwards, not just guarding before

Closing channels stops new contamination. It does nothing about telling you when something slipped through, so the last piece is a check that runs on the finished draft.

It compares the draft's vocabulary two ways: against what the piece is supposed to be about, and against the records the floor deliberately rejected. If the draft reads closer to the rejects than to the subject, something got through.

The interesting design constraint was making it shut up. Excluded is not the same as off-subject — plenty of legitimate writing touches a neighbouring idea, and a check that flags every adjacency is a check people turn off in a week. It abstains in six specific situations. Most of what that module does is stay quiet.

What moved

We keep a regression suite with committed fixtures, so this isn't a vibe.

Measured from the commit before this work to the commit after: subject fidelity went from 50% to 100% on the contamination suite. Half the fixtures were previously pulling in material from the wrong subject. None are now.

The other number I care about more: for a piece with no attached material and no stated brief, the composed prompt came out byte-identical to the one before any of this shipped. Every phase was built to degrade to exactly what it replaced when it has nothing to add. A context fix that changes every prompt is not a fix, it's a rewrite with a nice story attached.

What this means if you're building with an LLM

Three things I'd take to any team debugging "the AI keeps going off topic":

Count your channels, by reading the code. Not the ones you designed — the ones that exist. Ours had eight and I would have guessed four. The two that caused the most damage were both incidental: material assembled for one stage and reused by another because it was already in scope.

Find every unfloored retrieval. Anything that ranks without a floor will hand you its best guess on a subject it holds nothing about, and it will do so silently and confidently. Silent is the problem. A retrieval that can't return empty can't tell you it has nothing.

Resolve what the piece is about before you go looking. Whatever the user typed in the box is an instruction. The subject might be one dereference away, and if you skip that step every retrieval after it is aimed at the wrong target — accurately.

None of this is prompt engineering. You cannot phrase your way out of bad material. The model was never the problem; it was the only part of the system doing its job.

---

Zsper is an AI writing partner with memory — it learns your opinions, stories and voice, and writes articles, newsletters and LinkedIn posts that sound like you. Start free for 14 days, no card.