FreeLesson 1 of 36 min read
How a large language model predicts the next word
Learn how a language model turns text into tokens, assigns a probability to every possible next token, and builds whole answers one token at a time.
When a chatbot answers you, it can feel like it is thinking up a reply all at once. It is not. Under the hood, a large language model does one narrow job over and over: given some text, it predicts what comes next. Understanding this single step explains a lot about what chatbots do well and where they go wrong.
Step one: text becomes tokens
Models do not read letters or whole words directly. Text is split into tokens, which are chunks such as whole words, parts of words, or punctuation. A common word like "the" might be one token, while a rare word might be split into several pieces. Each token has an ID number, and the model works with these numbers.
Step two: a probability for every possible next token
Given the tokens so far, the model outputs a score for every token in its vocabulary, often tens of thousands of options. These raw scores are called logits. A function called softmax turns them into probabilities that are all positive and add up to 1.
Formally, the model estimates P(next token | all previous tokens). Intuitively, it is answering: "Based on patterns in the text I was trained on, how likely is each token to come next here?"
Example: completing "The cat sat on the ..."
Suppose (made-up numbers) the logits for three candidates are: mat 2.0, floor 1.5, sofa 1.0.
Softmax: exp(2.0) = 7.39, exp(1.5) = 4.48, exp(1.0) = 2.72. Sum = 14.59.
Probabilities: mat 0.51, floor 0.31, sofa 0.19.
The model does not "know" a cat sat on a mat. It has learned that "mat" often follows this kind of phrase.
Step three: choose a token, then repeat
Once the model has probabilities, a token must be picked. Greedy decoding always takes the most likely token. Sampling picks randomly in proportion to the probabilities, which makes output more varied. The chosen token is added to the text, and the whole process runs again to predict the following token. This loop is called autoregressive generation: each new token depends on everything before it, including tokens the model itself just wrote.
A setting called temperature T reshapes the probabilities by dividing the logits by T before softmax: softmax(z / T). Using the logits above, T = 0.5 gives mat 0.67, floor 0.24, sofa 0.09, so output becomes more predictable. T = 2 gives mat 0.42, floor 0.33, sofa 0.25, so output becomes more varied.
How the model learned these probabilities
Training uses a simple idea repeated on a huge amount of text:
- Take a real passage and hide the next token.
- Let the model predict a probability distribution.
- Measure the loss as -log(probability given to the true token). If the model gave "mat" 0.51, the loss is about 0.67; a higher probability means a lower loss.
- Adjust the model's parameters slightly to reduce that loss.
- Repeat across billions of examples until the predictions become good.
Chat assistants usually get further training afterwards, for example on example conversations and human feedback, so that their continuations look like helpful answers. But the core mechanism stays the same: next-token prediction.
Next time you read a chatbot reply, picture it being built one token at a time, each chosen from a probability distribution. In the next lesson we will open the box and see how the model decides which earlier words matter for each prediction: a mechanism called attention.
Key takeaways
- A language model splits text into tokens and predicts a probability for every possible next token.
- Text is generated by repeatedly choosing a token and feeding it back in as input.
- Training adjusts the model to give higher probability to the token that really came next.
Check your understanding
Written with AI help. Spotted a mistake? Tell us.