The long read
What happens when you talk to a chatbot
You type a question, and an answer appears a few words at a time. That steady trickle is a clue to how the whole system works. This piece follows your message from the moment you press send, in plain language, with deeper notes along the way for anyone who wants them.
Key ideas, in seven lines
- Text is split into tokens, and the model works with tokens, not letters.
- At each step the model gives a probability to every possible next token.
- A reply is built one token at a time, and each new token becomes part of the input for the next step.
- The context window is everything the model can see right now, including its own reply so far.
- Sampling and temperature decide which token is picked, which is why answers vary.
- Fluency comes from patterns in training text, so a fluent answer can still be false.
- Pretraining teaches prediction; later tuning, including learning from human feedback, shapes the model into an assistant.
Select any line to underline it in pencil or to write a note beside it. Your marks stay in this browser.
Your words become tokens
The program that writes a chatbot's replies is called a , or the model for short. The chatbot is the app you type into, and the model is the part that predicts the text. Before the model does anything with your message, the text is cut into small pieces called . A token is often a whole common word. Sometimes it is part of a word, a punctuation mark, or a word with the space in front of it attached. Long or unusual words tend to become several tokens, and short everyday words are often one.
Each token is then replaced by a number, its entry in the model's fixed vocabulary of pieces. From that point on, the model works only with numbers. It never sees your letters the way you do.
How big is a token? Usually a few characters: a short word, or part of a longer one. The size depends on the model and the language. Anthropic's documentation gives about 3.5 English characters per token for its earlier models, and says its newer models split the same text into about 30 percent more tokens. You rarely need to think about this, but tokens are the unit behind a model's length limits, and behind the price of using one through a developer service (access for software rather than people).
Figure B. How text becomes tokens
Illustration roughly how a tokenizer might split this; real tokenizers differ
48 characters, 6 words
A small dot marks a space carried at the front of a piece.
Two encodings from the same library split this sentence into 12 and 11 pieces.
Tokens also explain part of a well-known oddity. Ask a chatbot to count the letters in a word and it may slip, partly because the word reached the model as one or a few multi-letter pieces, not as a row of letters. The answer on why chatbots miscount letters looks at this more closely.
Going deeper
Many tokenizers are built with byte-pair encoding. You start from single characters, or from the 256 possible byte values, and repeatedly merge whichever adjacent pair appears most often in a body of training text, until the vocabulary reaches a chosen size. Working from bytes means any text can be encoded, with no "unknown word" gap. Each tokenizer is built from its own training text and vocabulary size and is fixed before the model is trained, so different models can split the same sentence differently.
The model predicts the next token
Now the central step. The model takes the full sequence of tokens so far and computes a score for every token in its vocabulary. Those scores are converted into probabilities that add up to 1. The result is a probability distribution, the model's estimate of how likely each possible piece is to come next.
Take the words "The cat sat on the". A model might give "mat" a high probability, "floor" a lower one, "sofa" lower still, and thousands of other tokens tiny ones. Figure A shows a toy version with six candidates.
Figure A. A toy next-word distribution
Illustration. These probabilities are invented to show the idea; a real model spreads probability over many thousands of tokens.
The cat sat on the
One token is chosen and added to the end of the sequence. Then the model runs again, now including that new token. Then again. A reply of 300 tokens means roughly 300 runs of the model, each producing a single piece. (In practice, work done on earlier tokens is stored and reused, so each run computes only the newest position.) Models that reason before answering write hidden reasoning tokens first, so a short reply can take many more runs. When an answer appears on screen a few words at a time, you are watching this loop.
That loop is the entire output mechanism. Predict a distribution, pick a token, append it, repeat. It sounds far too simple to produce a working piece of code or a careful summary of a long report. The ability lies in how good the predictions are, and that comes from training, which we will get to.
Going deeper
Most current chatbots are built on the Transformer architecture, introduced in 2017. Its central component, attention, lets each position in the sequence draw on information from earlier positions when the next scores are computed. "Next-token prediction" describes the output, and it leaves open what happens inside. In 2025, interpretability researchers at Anthropic reported that when one of their models wrote a rhyming couplet, it settled on candidate rhyme words before writing the second line, then wrote towards them. How widespread this kind of planning is remains an open research question.
The context window is its working view
What the model can take into account at any moment is called its . That includes instructions the app adds before your message (often called a ), the conversation so far, any document or image you attached, any reasoning the model writes before answering, and the reply it is in the middle of writing. Everything the model can use about your conversation sits in that one window, and you decide much of what goes in.
A chat app can add to the window too. Claude, for example, searches the web when a question calls for it and cites the pages it used (claude.com). A reply that shows its sources can be checked against them. A reply about recent events that shows none deserves more care.
The first surprise for many people is that the model's knowledge from training and its view of your conversation are separate. Chatting with a model does not change it while you talk, and the model is the same at the end of your conversation as at the start. (Some services may later use conversations as training data, which the answer on pasting work documents into a chatbot covers.) Memory features in some apps save notes for later chats; they do not change the model.
The second is that the window has a size limit, measured in tokens. Limits have grown a great deal, and some models now accept very long documents. A bigger window does not guarantee careful use of everything in it, though. A 2023 study found that the models it tested used information near the start or end of a long input more reliably than information in the middle (Liu et al., arxiv.org), and Anthropic's documentation still warns that accuracy tends to decline as the context grows (platform.claude.com).
Going deeper
Every new token is computed against the whole context, so long conversations cost more to run. When a chat nears its limit, apps deal with it in different ways. Some drop the earliest turns, and others summarize them. Either way, details from early in a very long conversation can fall out of view, which is one reason a fresh chat with a short summary sometimes works better.
How one token gets chosen
The model produces probabilities, but something still has to pick a token. The simplest rule is to always take the most likely one. Most chatbots instead , drawing a token at random, weighted by its probability. In Figure A, "mat" at 0.45 would be drawn about 45 times in 100, and "roof" at 0.05 about 5 times.
is a setting that reshapes the distribution before the draw. A low temperature sharpens it, so the favourite wins almost every time. A high temperature flattens it, so less likely tokens are chosen more often. In the toy example, temperature 0.2 lifts "mat" from 45% to 98.1%, while temperature 1.5 lowers it to 34.4% and raises "roof" to 7.9%. These changes affect the choice among tokens, not the knowledge stored in the model. More predictable or more varied text can still be wrong. Hugging Face explains the decoding choices.
Sampling is the main reason the same question can get different answers. Each reply is one path through an enormous number of possible continuations, and a different word early on can change everything that follows. Chat apps generally choose these settings for you. Developer tools often let you set them, but not for every model: as of November 2026, Anthropic's guide for Claude Opus 5.5 tells developers to leave them out (platform.claude.com).
Going deeper
Temperature divides the model's raw scores (logits) by T before they are turned into probabilities, which is the same as computing softmax(log p / T) from the original probabilities. Temperature 0 cannot literally be a divisor, so systems treat it as always taking the most likely token. Even so, Anthropic's documentation notes that outputs at temperature 0 are not fully deterministic in practice. Many systems also trim the long tail of unlikely tokens before drawing, for example with top-p (nucleus) sampling.
Why a fluent answer can still be wrong
A model learns what text tends to look like. Much of the time, text that looks right is right, because the training data held many correct statements about common things. Trouble starts where the two come apart.
When a fact was rare in the training data, absent from it, or came along after training ended, the model still produces the most plausible continuation it can. The result can be a smooth, confident sentence that is false. People call this , or confabulation. Fluent wording fits patterns the model learned. It does not prove the statement is true or that the exact sentence appeared in training data.
Errors are more likely with obscure people and places, exact figures and dates, quotations, references to papers or legal cases, recent events, long calculations, and questions with a false assumption built in ("Why did this film win the award?" when it did not).
Other situations make errors less likely. It helps when the source text is in the context and you ask the model to answer from it, when the model can use a tool such as search or a calculator, and when you invite it to say it is unsure. None of these removes the risk entirely.
Going deeper
Why not just say "I don't know"? A 2025 paper, "Why Language Models Hallucinate", argues that some errors are a natural statistical result of pretraining, and that widely used evaluations reward confident guessing over admitting uncertainty, much as an exam with no penalty for wrong answers rewards guessing. Separately, Anthropic's interpretability work found that one of its models has a default "can't answer" pathway that is switched off when the model recognizes something familiar. When that recognition misfires, the model answers anyway.
How it learned to do this
Training happens long before your conversation, in two broad stages.
comes first. The model starts as a very large set of adjustable numbers, called , set more or less at random. It is shown a vast amount of text and, at each position, asked to predict the next token. After each large batch of predictions, the parameters are nudged so the right tokens become slightly more likely. Repeated across an enormous collection of text, this produces a model that is very good at continuing documents. It is not yet an assistant. Given a question, a pretrained model may carry on with more questions, as if completing a list.
Further tuning comes next. The model is trained on examples of good responses to instructions, then refined using human judgement. In the method called , people rank alternative responses, a separate model learns to predict their preferences, and the chatbot is adjusted towards responses that this second model scores highly. Developers now add other techniques, such as feedback from another AI model guided by written principles (arxiv.org), and reinforcement learning on tasks with automatically checkable answers, like maths problems and code tests, which helps some models learn to reason before answering (arxiv.org). The full training recipe for most commercial models is not public.
Going deeper
In the 2022 paper that applied this approach to a model called InstructGPT, people preferred outputs from a tuned model with 1.3 billion parameters to those of the original GPT-3, which has 175 billion.
What this means when you use one
- Put what matters in the window. If the answer depends on a document, include it and ask the model to answer from it.
- Check what can be checked. Names, numbers, dates, quotes and references deserve a look at the original source.
- Ask twice when it matters. If two fresh answers disagree, treat both with care. Agreement is not proof, though, since a model can give the same wrong answer every time.
- Start fresh when a long chat drifts. A new conversation can remove irrelevant history. Carry over the goal, essential facts, constraints and sources. Anything important left out is no longer available to the model.
- Use tools for exact work. For counting, arithmetic or data, ask for step-by-step working, or for a calculator or code if the app offers one.
- Expect fluency, and judge accuracy separately.
Mini glossary
- Token
- a small piece of text, often a word or part of a word, that a model reads and writes.
- Probability distribution
- a set of likelihoods for every possible next token, adding up to 1.
- Context window
- everything the model can take into account at once, measured in tokens.
- Temperature
- a setting that makes the choice of next token more predictable (low) or more varied (high).
- Pretraining
- the first and largest stage of training, in which the model learns to predict the next token across a vast amount of text.
- RLHF
- reinforcement learning from human feedback, a tuning method that uses people's rankings of responses.
Sources and further reading
All fetched on 4 October 2026, unless an entry gives a later date.
- Vaswani et al., "Attention Is All You Need" (2017), the paper that introduced the Transformer. arxiv.org/abs/1706.03762
- Anthropic, Glossary (token size by model, temperature 0 not fully deterministic, pretraining, RLHF), fetched again 11 November 2026. platform.claude.com/docs/en/about-claude/glossary
- Hugging Face LLM Course, "Byte-Pair Encoding tokenization". huggingface.co/learn/llm-course/chapter6/5
- 3Blue1Brown, "Transformers, the tech behind LLMs" (visual explainer covering tokens, embeddings, softmax and temperature). 3blue1brown.com/lessons/gpt
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023). arxiv.org/abs/2307.03172
- Ouyang et al., "Training language models to follow instructions with human feedback" (2022), the InstructGPT paper. arxiv.org/abs/2203.02155
- Kalai et al., "Why Language Models Hallucinate" (2025). arxiv.org/abs/2509.04664
- Anthropic, "Tracing the thoughts of a large language model" (2025), the rhyme-planning and "can't answer" pathway findings. anthropic.com/research/tracing-thoughts-language-model
- Anthropic, "Context windows" (what the window holds, including reasoning, and accuracy declining as context grows, called "context rot"). platform.claude.com/docs/en/build-with-claude/context-windows
- Bai et al., "Constitutional AI: Harmlessness from AI Feedback" (2022), training with AI feedback guided by a list of principles. arxiv.org/abs/2212.08073
- DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" (2025). arxiv.org/abs/2501.12948
- Anthropic, "Claude Opus 5.5 migration guide" (sampling settings left out of requests), fetched 11 November 2026. platform.claude.com/docs/en/models/opus-5-5/migration-guide
- Anthropic, "Claude can now search the web" (when Claude searches, and its citations), fetched 11 November 2026. claude.com/blog/web-search
Changes. 11 November 2026: the size of a token and the note on temperature settings now match Anthropic's current documentation, a paragraph on chat apps that search the web was added, and the suggested next steps changed.
New to this? Try the exercise Test it on what you know, then read How do I know if an answer is right?
Want the machinery? Read what a GPU needs to run a model, why answers take time, and how model servers work.
Wondering about something this piece did not answer?