It sounds like this“It already has the whole answer and just types it out slowly for effect.”
Picturing the reply as written in full, then typed out
Why it feels right
Replies arrive fluent, well organised and apparently planned, and the text streams out at a steady pace like someone typing up a finished draft. There is also a grain of truth: research that looks inside models has found them preparing for words several steps ahead, such as a rhyme at the end of a line. But preparing is not the same as having written. Each word is committed one at a time.
Why it’s wrong
Each token is chosen only after the tokens before it exist. Its internal state can anticipate where it is heading, but the words are committed one at a time and each one changes what follows.
A worked example
Ask a model for 17 times 24 and make it answer in a single token's worth of thinking. It has to produce the digits of the answer straight away, and for harder sums it is often wrong.
Now ask it to work it out. It writes '17 times 20 is 340. 17 times 4 is 68. 340 plus 68 is 408.' Those lines are now part of the text it reads, so by the time it reaches the answer, the token '408' scores far higher than any other. The working is not a performance for the reader. It is where the model does the work, because the only place it can keep intermediate results is in the text it has already written.
The flip side is that a wrong start is hard to undo. A confident-sounding first sentence becomes text the rest of the answer has to fit.
How to catch yourself
- You believe the streaming of a reply is cosmetic and the answer is already finished.
- You think asking it to 'think step by step' only changes how the answer looks.
- You expect it to notice mid-reply that its first sentence was wrong.
What to hold instead
Each token is chosen only after the ones before it exist, and then it becomes part of the text the model reads next. What it has written steers what comes after. That is why asking for working before the answer helps, and why asking for the verdict first can lock in a guess.
Where you’ll meet it
- 'Think step by step' and other prompts that ask for working before the answer.
- Replies that contradict themselves partway through.
- Why a long answer can start well and wander: each choice constrains the next.
Questions people ask
- Does AI plan ahead when it writes?
- In part. Studies of what happens inside models have found them lining up a rhyme, or a word several tokens ahead. But the words themselves are chosen one at a time, and what has been written changes what comes next.
- Why does asking an AI to explain step by step improve its answers?
- Because the steps become part of the text it reads as it continues. Intermediate results it writes down are available when it gets to the answer, where otherwise it would have to produce the answer in one go.