AI Dojo Back to the typewriterThe typewriter

The long read

Why is my model slow?

9 min read
First published 4 November 2026.

A reply can feel slow in two different ways: nothing appears for a while, or the words arrive slowly once it starts. Those symptoms point to different work. This piece follows a request through a running model and gives each speed measurement a plain meaning. If inference, GPU memory or the KV cache are new, start with Will this model fit on my GPU?

Key ideas, in seven lines
  • Prefill processes the prompt; decode continues the answer, and a request may queue before either.
  • Time to first token (TTFT) measures the wait until output starts. Lower means an earlier start; it does not guarantee the full answer finishes sooner.
  • Time per output token (TPOT) measures the average time per additional token after the first. Lower means a faster average stream; it excludes the initial wait.
  • Throughput counts total work per second; latency describes one request's wait.
  • An output cap limits new tokens; a stop reason helps explain why an answer ended.
  • Continuous batching admits new work as earlier requests finish, within resource limits.
  • Prefix caching reuses matching prompt computation; the answer still has to be generated.

Select any line to underline it in pencil or to write a note beside it. Your marks stay in this browser.

Before and after the first token

Inference means using a trained model to produce an answer. For the usual text-generation loop, the work has two stages. Prefill processes the prompt. Decode generates the continuation, using the prompt and the output so far. A request can also spend time waiting for capacity before model work starts. The vLLM request lifecycle records queue, prefill and decode intervals separately.

  1. The request arrives and may wait in a queue.
  2. Prefill processes the input tokens and produces the information needed to start answering.
  3. Decode continues the answer until a stopping condition is reached.

This gives you two useful questions before changing any settings: “Which part of the wait grew?” and “Did the input or the output get longer?” Neither question needs a new server. They need measurements from the request you actually care about.

How long until it starts?

Time to first token (TTFT) measures the initial wait. From a client's point of view, it runs from sending a request to receiving the first output token. Lower TTFT means output starts sooner. It does not say how quickly the rest arrives. NVIDIA's metric definitions make that boundary explicit.

The client is the software sending the request. Its clock can include network travel and server waiting as well as model computation. A server's internal timer can start later. Comparing client time on one system with internal model time on another can make a faster-looking number meaningless. Use the same start and finish events.

Figure 1. One request, three timestamps

Illustration Invented timings for an answer of 11 output tokens.

Request sent
0.0 seconds
First token received
1.0 seconds
Last token received
3.0 seconds
The initial wait is 1 second. The complete answer takes 3 seconds. Both describe this same request.

How fast does the rest arrive?

Time per output token (TPOT) is the average time for each additional output token after the first. Lower TPOT means a faster average stream once output has begun. It leaves out the initial wait. For an answer with more than one token, calculate it this way:

(Total time − TTFT) ÷ (number of output tokens − 1).

In Figure 1, the remaining 10 tokens arrive over 2 seconds. TPOT is 2 ÷ 10 = 0.2 seconds per token, or 200 milliseconds per token. The reciprocal is 5 tokens per second over that part of this request. These are illustrative calculations, not a measured model speed.

Going deeper: an average can hide pauses

An average does not show whether output arrived steadily or in bursts. Inter-token latency is often measured between streamed output events; an event can contain several tokens. Current vLLM documentation distinguishes that event-level measure from per-request TPOT. Check the tool's definition, especially with methods that emit multiple tokens together. A one-token reply has no interval after its first token, so tools may exclude it from TPOT summaries or record a special value.

More work per second, or less waiting?

Latency is the time one request takes. Throughput is work completed across a system per unit of time. A server might report output tokens per second or completed requests per second. State which one: input tokens, output tokens and requests count different things. NVIDIA's metrics guide defines both token and request rates.

Higher throughput does not guarantee less waiting for one person. Processing requests together, called batching, can increase total throughput while using more memory or increasing latency (Hugging Face).

Illustration: suppose one setup completes one 100-token reply in 5 seconds. That is 20 output tokens per second over the whole run. Another setup completes four 100-token replies together in 10 seconds. That is 40 output tokens per second across the server, but each person waits 10 seconds. The second setup has twice the total throughput and twice the individual completion time. This comparison illustrates the measures; it does not predict what a real batch will do.

Choose the measure that matches the job. For a chat interface, the start delay and the pace of the stream both matter. For a large overnight job, total completed work may matter more. For either, include failures in the report rather than counting only successful fast replies.

Why did the answer stop early?

A short answer is not automatically an efficient answer. Generation can stop because the model emits an end-of-sequence token, a token meaning it is finished, or because a limit or another stopping rule is reached. max_new_tokens in Transformers caps the number of newly generated tokens. Raising it permits more output; it does not force the model to use the allowance (generation controls).

Context limit
How much token context the model and serving setup allow for a request. In the ordinary decoder-only setup, the input and generated continuation must fit that total allowance.
Output cap
The maximum number of new tokens allowed for the answer. It is not the length of the prompt.
Stop reason
The server's explanation of why generation ended, such as a stop token, a length limit or an aborted request. Inspect it alongside the answer.

Illustration: a request with a 2,000-token input and a 100-token output cap can stop after 100 new tokens even if its configured context limit is 8,000. The unused context room does not override the smaller output cap. A length-limited JSON answer might end halfway through a field, so comparing its time with a complete answer would reward the wrong behaviour.

Exact parameter names and finish reasons depend on the API. vLLM exposes a maximum combined prompt-and-output length and records completion reasons in its engine configuration and production metrics. Check those limits before treating a quick cutoff as an improvement.

Keep working when one answer finishes

A batch is a group of requests processed together. Their answers rarely all have the same length. If the server waits for the whole group to finish before admitting anyone new, a short completed answer leaves capacity unused while a long answer continues.

Continuous batching revisits the group at generation steps. Completed requests leave and new requests can join while other answers continue. That can keep the hardware busier without waiting for the longest answer in the original group (Hugging Face).

Illustration: A finishes after 20 output tokens and B needs 200. With continuous batching, C can take A's place while B continues. This describes admission to shared work, not a promise that C never waits. Memory and computation still limit how much the server can admit. Use both throughput and latency to judge the result.

Reuse the beginning, then write a new answer

The key-value (KV) cache stores attention calculations for earlier tokens so later tokens can reuse them. The GPU memory piece explains why this cache takes space. Prefix caching goes further: a new request can reuse saved work for the same starting token sequence as an earlier request.

Imagine two questions about the same long document. If the document and preceding instructions form a matching cached prefix, the second request can skip processing that shared part again. It still needs to process its new question and generate its answer. A matching passage somewhere in the middle of a different prompt is not the same as a matching prefix.

The benefit is reduced prompt-processing work. Prefix caching does not itself speed up generation of each new token. It helps less when prompts do not share a reusable beginning or when most of the time is spent generating long answers. Cache availability and the engine's matching rules also matter (vLLM's explanation and limits).

That is enough to interpret the main levers: a shorter initial wait, a faster ongoing stream, or more total work completed. The next piece explains the serving engines that put these mechanisms together.

Review the key ideas

Next: the software that serves the model

How do we serve a model?