Why the Same Question Gets a Different Answer

Open a new chat and ask your last important question again, word for word.

You may get a different answer. Earlier you asked whether you could claim an expense, and the AI said yes. This time it says "it depends", and then, roughly, no.

It's the same question in the same tool, five minutes apart. Most people decide the AI is broken, or that it can't be trusted, but both guesses miss the real cause.

The cause is dice, and once you see them, the dice become one of the best checks you have.

A loaded die for every word

An AI chatbot writes its reply one word at a time, and at each step it has several words to choose from, each with its own odds. Take your expense question. "Yes" might be the likeliest first word, at 60 per cent, while "It" (as in "it depends") might be at 30, with other words making up the rest.

The AI doesn't take the favourite every time. Instead it rolls a loaded die, weighted by those odds, because always taking the top word would make it sound like a robot repeating itself.

The first roll steers everything after it. "Yes" leads to a confident paragraph about claiming, and "It" leads to a paragraph of conditions. So you get two replies from the same model, the AI system behind the chat. Both are written with the same confidence, and neither knows the other exists.

For most questions the die is so heavily loaded that it lands the same way every time. Ask when the Great Fire of London was and you get "1666" on every roll, because no die can shift a word with 95 per cent odds.

The rolls only vary where the model is unsure, which is where two answers sit close together in its head.

What the roll tells you

A different answer tells you the model wasn't sure. That's all, and that's a lot.

The question sits on a fault line. Maybe the rule is complicated or has changed, or it depends on something you didn't say, or the model only half-knows it. Any of those is a reason not to act on either reply.

What a different answer can't tell you is which answer is right. People trust the second because it came later, or the first because it sounded surer, but the die doesn't know.

What you've learned is that this question needs settling with a source, a person or a search.

A different answer doesn't mean the AI is broken. It's your cue to check.

Rolling on purpose

For anything that matters, ask the same question in three fresh chats and line the answers up.

If all three match, the model is sure, but that doesn't make the model right. Three rolls of a die loaded the wrong way all land wrong together. Matching answers measure confidence, and for truth you need How Do You Know?

Even so, a match means you can check one source instead of three.

If all three differ, use none of them and go to the source. Then notice what changed between the replies. Usually it's the condition you didn't state, such as "it depends whether you're self-employed".

If two agree and one doesn't, read the odd one out. The odd one is often the careful, hedging answer, and the hedge is often right.

The same trick works on drafts, so try three versions of the difficult email or three plans for the week. Where they agree, you've found the solid part, and where they differ, the decision is yours to make.

No second opinion is cheaper, and it takes thirty seconds.

Two things can spoil the roll. First, memory and Projects can make three answers match, because these features let the AI remember what it told you last time.

So use fresh chats with memory switched off, or use a temporary chat. Second, some tools and settings turn the dice down to give more consistent answers. That hides the warning without fixing the problem it warned you about.

Whether a question that gets two answers belongs with a chatbot at all is your call to make.

Before you act on the next answer that matters this week, ask the same question twice more in fresh chats.