Why It Makes Things Up

When an AI invents a fact, people in the field call it a "hallucination." It is a strange word for a piece of software, but it stuck because the behavior feels like one: the AI produces something detailed, specific, and completely untrue, with no apparent awareness that it did anything wrong.

To stop being surprised by this, you need one mental model. An AI chatbot is, at its core, a very sophisticated guesser of what words come next. You give it some text; it predicts the most plausible continuation, word by word, based on patterns it absorbed from an enormous amount of writing. That is the whole job. It is astonishingly good at it - good enough to write code, explain tax rules, and draft a wedding toast. But notice what that job does not include: checking whether the plausible-sounding thing it is about to say is actually true.

Plausible is not the same as true

This is the crux. The AI is optimized to produce text that sounds right. Most of the time, text that sounds right also is right, because correct information is common in what it learned from. So it gets a huge amount correct, and you start to trust it. Then it hits a gap - a question where it does not have a solid pattern to draw on - and it does the only thing it knows how to do: it generates the most plausible-sounding answer anyway.

The result is fluent and wrong at the same time. Ask it for a book on a niche topic and it may give you a real-sounding title by a real author that does not exist. Ask for a legal citation and it can produce a case name, a court, and a year, all formatted perfectly, all invented. The famous real-world example: in 2023, lawyers submitted a court brief full of fake cases an AI had generated, complete with fake quotes. The cases looked exactly like real ones. They were not, and the lawyers were sanctioned.

The reason it looks so convincing is the same reason the true answers look convincing - it is using the identical machinery for both. A fake citation is built from the same patterns as a real one. There is no separate "honesty mode" that kicks in for facts.

There is no inner fact-checker

Here is the part that trips people up most. When the AI tells you something false, it is not lying, because lying requires knowing the truth and choosing to hide it. The AI does not have a stored database of verified facts it consults and then decides whether to share. It has patterns. When you ask a question, it is not looking something up - it is composing an answer that fits the shape of a good answer.

So "Are you sure?" is a weak defense. If you push back, the AI will often apologize and either change its answer or double down - not because it re-checked against reality, but because your pushback changed the pattern of plausible next words. Sometimes it corrects a genuine mistake. Sometimes it "corrects" something that was right. You cannot tell which from the tone, because the tone is always confident.

This also explains why confidence carries no information. A human expert usually signals uncertainty - they hedge, they slow down, they say "I'd want to double-check that." The AI's fluency is constant. It is not connected to how solid the underlying answer is. A wild guess and a rock-solid fact come out in the same even voice.

What this means for you

None of this makes AI useless - far from it. It means you should hold its output the way you would hold a confident first draft from a sharp but unreliable colleague: a great starting point, worth real attention, and never something you forward without reading.

A few practical takeaways follow directly from the mental model:

• Trust drops as the question gets more specific and more obscure. Common, well-trodden topics are mostly fine. Exact names, dates, numbers, quotes, and citations are where invention creeps in.
• More detail is not more reliable. A made-up answer can be richly detailed. Specificity is not evidence of truth - sometimes it is the opposite, a sign the model is filling a gap.
• Confidence is not a signal. Stop reading the tone as reassurance. It is the same tone for everything.

Some tools now reduce this by actually searching the web or your documents and quoting what they find, which helps a lot - but even then they can misread or misattribute a source. The underlying tendency to produce plausible text never fully goes away.

Once you internalize why it makes things up, the fix becomes obvious: do not ask the AI to be the source of truth. Ask it to do the work, and keep the truth-checking on your side. That is the next phase.