It writes essays, code and poems. It feels like it understands. But at its core, ChatGPT does one surprisingly simple thing, billions of times: it guesses the next piece of text.
It predicts the next token
Your message is cut into tokens — chunks like ␣Paris, ing or .. The model looks at all tokens so far and outputs a probability for every possible next token. One is picked, added to the text, and the whole process repeats. Word by word, an answer appears — that's why it “types”.
Play with probabilities
Pick a prompt, move the temperature, then generate 20 times. Same question, different answers.
What the model sees: 5 tokens
It's not retrieving an answer. It's writing the most plausible continuation.
How it learned
- Pre-training: the model read a huge amount of text and practised predicting the next token, adjusting billions of internal numbers (parameters) each time it was wrong.
- Fine-tuning: it was trained on example conversations to follow instructions.
- Human feedback: people ranked answers, teaching it to be more helpful, honest and safe.
The transformer trick
The architecture behind it — the transformer — uses “attention” to decide which earlier tokens matter for the next one. In “The trophy didn't fit in the suitcase because it was too big”, attention links “it” to “trophy”.
Why it makes things up
The model is rewarded for plausible text. Ask about an obscure book and the most plausible continuation is a confident, well-formatted summary — even if the book doesn't exist.
Quick check
Which reduces hallucinations the most?
Prompting, explained by the model
Since it continues text, better context = better continuation:
- Give a role and audience: “You are a tax advisor explaining to a freelancer in Morocco…”.
- Give examples of the output you want — it will match the pattern.
- Give the data instead of hoping it knows.
- Ask for steps before the final answer on complex problems.