Why AI Makes Things Up (And Sounds So Sure While It Does)

How can AI recommend a book that doesn't exist, and not even blink?

There was a title, an author, a year and a one-line summary of the argument. You went to order the book and found nothing, because there was nothing. The author is real, but the book isn't.

Everyone who uses these tools has a story like this. It might be the court case that was never heard, the study with no authors or the quote nobody said. Nothing does more to make people stop trusting AI, or, worse, to trust AI anyway and get caught out.

This won't be fixed next month, because it comes straight out of how AI works. That means you can learn when to expect it.

What is the AI doing when it answers me?

An AI chatbot (a program you type to and it types back) has read an enormous amount of text. From that, it learned which word tends to come next. That's the whole mechanism, and at no point does it look your answer up in a store of facts.

Instead the AI writes the reply one word at a time. It picks each word because that word is the most likely next step after everything before it.

Most of the time that produces the truth, because the truth is what usually came next in everything the AI read. Ask when the Great Fire of London was, and "1666" is by far the most likely thing to say.

But ask for a book on a narrow topic by a real author, and now the most likely next step is a title that author might have written. Nothing checks whether the book exists. The sentence came out well formed, and that was the only test the AI was built to pass.

Think of a well-read friend who answers every question instantly and never once says "I don't know". They're brilliant company, but they're a terrible source when you need a reference you can check.

Then why does AI sound so sure?

Because the tone and the truth come from different places.

The tone comes from what the AI learned from, which means reference books, articles, textbooks, and forum answers from people who sound certain. Most written answers sound confident and smooth, so that's how the AI writes, for true things and made-up ones alike.

The truth, when it's there, comes from whether the right answer happened to be the most likely one. Nothing links the tone to the truth. A made-up book gets the same calm sentence as a real one, because both sentences are built the same way.

You can't treat confidence as a clue, but oddly, you can treat hesitation as one. When the AI says "I'm not certain, but", its training has taught it that people disagree here, and that's often worth more than a clean answer.

When should I expect it?

Not evenly. Five places account for most of the made-up answers, and they all share the same shape. The real answer is rare, specific or recent, so a likely-sounding one is easier to produce.

1. References to books, papers, court cases and quotes are where you're most likely to catch AI inventing. The format is easy to copy, but the exact item is hard to remember.
2. Numbers and dates (prices, statistics, deadlines, head counts) come out as precise figures. That's because answers usually contain precise figures, whether or not the AI really has one.
3. Anything recent may be past the AI's cut-off date, which is the date its reading material stops. After that date, the AI is guessing from older patterns. Ask about this year's rules and you may get last year's, stated as if they're current.
4. Niche topics like your local council's bin policy, a small company or an unusual hobby have less text for the AI to learn from. Less text leaves more gaps to fill.
5. Long chats are the fifth. Deep into a conversation, the AI loses track of what you said and fills the gaps with what you probably said. There's more on the long-chat page.

A sixth is down to you, because AI makes things up more when you've shown which answer you want. Ask "isn't it true that…" and the AI will often agree, since agreeing is the most likely way to continue a leading question.

Is the AI lying to me?

No. Lying needs someone who knows the truth and says something else, and there's nobody in there who knows the book doesn't exist. Calling it a lie makes the AI sound more cunning than it is, and it makes you angrier than is useful.

And this isn't a reason to stop. A tool that invents in five known places and rarely elsewhere is one you can work with. Think of a friend who's brilliant on the big picture and hopeless with dates.

You don't drop that friend. You check the dates.

Won't the next version fix it?

The tools are getting better, and they make things up a lot less than in the early days. But the rate hasn't reached zero, and given how the mechanism works, I doubt it can.

The "next word" picture is a simplified version, and the people who build these tools would add extra detail. Even so, the picture predicts when the tool will let you down, which is what you need.

What can I do about it?

Turn web search on for anything with a date, a price or a name in it. Search changes the job from remembering to looking up. Even then the AI can misread the page, but at least it's reading a page.

Ask how the AI knows. This follow-up question splits an answer into what's sourced, what's estimated and what's guessed. Asking takes ten seconds, every time it matters. The question has its own page.

Put the material in front of the AI. Paste in the contract or upload the report, because an AI reading a document invents far less than one working from memory. The most likely next step becomes "what the document says".

For the same reason, the paperwork page insists on the whole PDF.

Tell the AI to admit when it doesn't know. Add a standing instruction to your settings, one that applies to every chat. "If you're not sure, say so. Never invent a source, a number or a quote."

That won't make the tool perfect, but it clearly helps.

Check the one claim everything rests on. You can't check everything, and you don't need to. Whether the likely-sounding answer is true, in this case, today, is your call, and no setting makes it for you.

Find the single claim the answer stands on and check that one yourself. If it holds, trust the rest for now. If it doesn't, trust none of it.