Explainer
Token
The small pieces of text a language model reads and writes.
Language models, the programs that write a chatbot's replies, do not read letters or whole words directly. A program called a tokenizer first cuts text into tokens. Common words are often a single token, rarer words are split into several pieces, and spaces and punctuation get folded in too. Each token maps to a number, and the model works only with those numbers. Tokens matter in practice because they are the unit for a model's length limit, for the price of a developer service (access for software rather than people) and for the speed of a reply. They also help explain some odd mistakes, such as miscounting the letters in a word that reached the model as multi-letter pieces.
Going deeper
Most tokenizers are built with a method like byte-pair encoding, which repeatedly merges the most frequent adjacent pieces in a training text, so the vocabulary reflects whatever text the tokenizer was built from (Hugging Face LLM Course, huggingface.co).
Read next: Context window