One simple task, repeated
Large language models can answer questions, summarize documents, translate and write code, which can make them seem like a collection of separate skills. At the core of the training, though, there is a single, simple task: given some text, predict what comes next.
If the text so far is “The capital of France is”, a good prediction for the next word is “Paris”. If it is “She opened the door and”, many continuations are plausible, and a good model assigns some likelihood to each rather than committing to one. The output of the model at each step is not a single word but a set of probabilities across all the pieces of text it knows.
What “predict” means here
It helps to be precise. The model does not look up an answer. It computes, from the text it has been given, a score for every possible next piece. Those scores are turned into probabilities, and one piece is chosen, usually with some controlled randomness. That piece is appended to the text, and the process repeats to choose the one after it.
This is why the same question can lead to different answers on different tries. Choosing the most probable option every time tends to give dull and repetitive text, so tools commonly allow some variety. The pieces being predicted are usually fragments of words rather than whole words, which is the subject of the article on text tokens.
How training works
Training starts with a very large collection of text. The text is cut into passages, and for each position in a passage the model is asked to guess the next piece, given everything before it. Then:
- The model makes a guess.
- The guess is compared with what actually came next.
- The difference is measured as an error.
- Millions or billions of internal numbers, often called parameters, are adjusted slightly in the direction that would have reduced the error.
Repeated over a vast quantity of text, this gradually shapes the parameters. At the start they are essentially random, and the model’s output is gibberish. Over time, it learns spelling, grammar, common phrases, and then more subtle regularities.
The internal machinery is built from neural networks, which process the surrounding text and weighs which earlier parts are most relevant to the next prediction. The specifics are technical, but the key point is that the system adjusts itself from examples rather than being programmed with grammar rules.
The scale involved is large. Training typically needs specialized chips and a great deal of computing time, which is part of why graphics processors became so important to this field.
What gets learned along the way
To predict text well, a model has to pick up many things as a side effect. Consider the sentence “After the match, the winning team celebrated, and the ___ team went home.” To fill the blank sensibly, the model benefits from capturing that “losing” fits better than “winning”, which requires something like tracking the relationship between two teams.
Across enough examples, models tend to absorb:
- grammar and style in many languages
- common facts that appear frequently in text
- the typical structure of documents such as emails, recipes, contracts and code
- patterns of reasoning that people write down, such as worked examples
None of this is stored as a list of facts. It is spread across the parameters in a form that nobody can read directly, which is part of why researchers work on explainable AI.
What it does not mean
The phrase “just predicting the next word” can be misleading in either direction. It does not mean the system is trivial: predicting text well across all topics demands a lot of internalized structure. But it also does not mean the system understands in the way a person does, or that its predictions are checked against reality.
A language model is rewarded in training for producing text that resembles its examples. If a false statement would be a likely continuation, the model has no inbuilt reason to avoid it. This is the root of AI hallucinations and why fluent writing is not evidence of accuracy.
Other consequences:
- Knowledge has a cutoff. The model only knows what appeared in its training text, up to the point training ended, unless it is connected to other tools.
- Frequent things are easier. Common topics are handled better than rare ones, which is part of why the choice of training data matters so much.
- Wording affects the prediction. The text you supply is the starting point for the prediction, so changes in phrasing change what is likely to follow, a topic covered under prompt wording.
From predictor to assistant
A model trained only to continue text is not yet a helpful assistant. If you type a question, it might continue with more questions, as in a list on a web page, rather than an answer. To make it behave like a helper, developers usually add further training on examples of good conversations and on human ratings of responses. This is a form of model fine-tuning, and it shapes tone, safety behavior and willingness to follow instructions on top of the base ability to predict text.
Why this explanation is useful
Knowing the mechanism helps set expectations. These systems are strong at producing plausible, well-formed text and weaker at guaranteed accuracy. They can be a useful aid for drafting, explaining and brainstorming, and they deserve checking where facts matter. The details of the field change quickly, and what the largest systems can do is a moving target, so treat any specific claim about capability with caution.