Computers need numbers
A language model is, underneath, a very large mathematical system. It cannot take in a sentence as people do. Text has to be converted into numbers first, and the way that is done affects what the model finds easy or hard. The conversion begins with splitting the text into pieces called tokens.
A token is a chunk of text that the model treats as one unit. Depending on the system, a token might be a whole common word, part of a longer word, a single punctuation mark, or even a single character. Each distinct token in the system’s vocabulary has an identifying number, and that number is what the model actually works with.
Why not just use words, or letters?
There are two obvious alternatives, and each has problems.
Whole words. A vocabulary of every word would need to be enormous, since languages have inflections, compounds, names, typos and newly coined terms. Any word not in the list would be impossible to represent. Rare words would also be seen too seldom in training for the model to learn much about them.
Single letters. A tiny vocabulary of characters can represent anything, but every sentence becomes very long. Longer sequences take more computing effort to process, and the model would need to learn to assemble letters into meaningful units on its own.
Tokens sit between these extremes. Common words get their own token, which keeps frequent text short, while rare words are broken into smaller recognizable pieces. A hypothetical split might turn “unbelievably” into pieces like “un”, “believ” and “ably”. A word never seen before can still be represented by assembling familiar parts, so nothing is completely out of vocabulary.
How the pieces are chosen
The list of tokens is typically built by examining a large body of text and finding which sequences of characters occur most often. Frequent sequences are merged into single tokens, and the process continues until the vocabulary reaches a chosen size, often in the tens of thousands or more. The exact vocabulary varies from one system to another, so the same sentence can be split differently by different models.
Spaces and capitalization are often part of the token, which means that “apple”, “ apple” and “Apple” may be different tokens. That is one of several quirks that look strange to a human reader but are natural for the system.
What happens next
After splitting, each token is replaced with its number and then with a longer list of numbers, called an embedding, that the model has learned to associate with that token. These are the inputs to the neural networks that process the text. The model’s output is then a score for each token in its vocabulary, which is converted to probabilities and used to pick the next token, as outlined in the article on how large language models learn to predict words. The chosen token is turned back into text, and the process repeats.
Why this matters in practice
Knowing about tokens explains a number of everyday observations.
- Length limits are counted in tokens. A model can only consider a limited amount of text at once, often called its context window, and that limit is measured in tokens, not words. English text typically works out to somewhat more tokens than words, but the exact ratio varies with the text and the system.
- Pricing and speed often track tokens. Services commonly measure usage in tokens, so longer prompts and answers generally cost more and take longer.
- Spelling and letter tasks can trip models up. Because the model sees chunks rather than letters, questions like “how many times does a particular letter appear in this word?” can go wrong. The letters are not directly visible to it.
- Arithmetic can be shaky. Numbers are also split into pieces in ways that do not line up neatly with place value, which is one reason that models can make errors on calculations unless they are given a calculator-like tool.
- Some languages cost more. Tokenizers built mostly from English text may split other languages into more, smaller pieces, which means the same content uses more of the length limit. This is one effect of the balance issues described under training data.
- Odd strings behave oddly. Unusual text, such as long random codes, may break into many tokens and be handled less reliably.
Not just for text
The same idea applies beyond writing. Images can be divided into patches, and audio into short slices, which are then treated much like tokens. This lets one general approach handle many kinds of data. For speech, the step from sound to text has its own considerations, covered in the article on speech recognition.
A few caveats
Tokenization is an engineering choice, not a law of language. Designers keep experimenting with other approaches, including working directly on characters or bytes, and the details may change. Likewise, the figures people quote for how many words fit in a given number of tokens are rough averages, not guarantees.
The takeaway is simple. Tokens are the pieces of text a model actually sees, chosen as a compromise between a vocabulary that is too large and sequences that are too long. Understanding them clears up why length limits, costs and certain odd mistakes exist, without changing the larger point that fluent output still needs checking.