The long read
How do we serve a model?
Downloading model weights is only part of running a service. Something has to load those weights, accept requests, share hardware and return answers. That software is the serving engine. This piece explains its job, then introduces vLLM and SGLang. Read Will this model fit on my GPU? for the hardware basics and Why is my model slow? for batching, caching and speed measurements.
Key ideas, in six lines
- The model supplies learned weights; the serving engine runs it and manages requests and resources.
- Multiple GPUs need an execution plan; their memory and speed do not combine automatically.
- vLLM is serving software with batching and cache management, including PagedAttention.
- SGLang is serving software whose RadixAttention reuses cached work for shared prefixes.
- A fair engine comparison controls the workload and reports speed, failures and task quality together.
- A schema constrains output structure; it does not prove the values answer the task correctly.
Select any line to underline it in pencil or to write a note beside it. Your marks stay in this browser.
The model and the engine do different jobs
A trained model consists of an architecture, the rules for its computation, and learned numbers called weights. A serving engine is the software that loads and executes models, accepts requests and manages shared resources. vLLM is one such engine. Changing the engine is not the same operation as training new weights.
- An application sends a request through an API, a defined interface for software to communicate.
- The serving engine prepares and schedules the request, then runs the model on available hardware.
- The engine returns the output, either as it is generated or after it completes.
The engine can support features such as batching and prefix caching that change how efficiently work is done. Compatibility still matters: support for a model architecture, hardware device and number format has to line up. “It loads” is the beginning of a test, not evidence that every feature you need works. The supported-model list is a better starting point than a generic speed claim.
Two GPUs need a plan
Two GPUs do not automatically act like one larger GPU. Software must decide how to divide the model or requests, and the devices may need to exchange intermediate results. More total memory is useful only when the execution plan can use it.
- Tensor parallelism
- Split work within model layers across devices. Each GPU computes part of the result, and they exchange information to continue.
- Pipeline parallelism
- Put different groups of layers on different devices. Intermediate results pass between those groups.
- Data parallel serving
- Run model replicas and give them different requests. This is a way to serve more work when one model already fits per replica.
Hugging Face's tensor-parallel guide explains the within-layer split and its communication requirement. The vLLM scaling guide covers deployment choices; its data-parallel guide describes replicas.
Illustration: two GPUs with 24 GB each do not guarantee that a 40 GB set of weights will load. The engine needs a supported way to split it, and each device still needs space for its share of working memory. Nor does adding a second GPU guarantee half the latency: communication and the distribution of work affect the result. Start by deciding whether the problem is fitting one model or serving more requests.
What vLLM does
vLLM is an open-source inference and serving library. Its engine combines model execution with request scheduling, continuous batching and cache management. It can expose an API so an application can send prompts without managing every generation step itself (vLLM documentation).
Its PagedAttention work addresses how attention-cache memory is allocated. Rather than requiring one large contiguous reservation for each sequence, it manages the cache in blocks. That reduces wasted reserved space and supports sharing cached work. It changes memory management, not what a token means (the original PagedAttention paper).
A useful first question is “Does this version support my exact model, hardware and required output features?” Only then ask whether its latency and throughput suit the workload. The project name alone cannot answer either question.
What SGLang does
SGLang is an open-source framework for running and serving language and multimodal models. Multimodal means the model can work with more than one kind of input or output, such as text and images. SGLang supports prefix caching and execution across multiple GPUs (SGLang documentation).
RadixAttention is its approach to organizing and reusing cached attention calculations for shared token prefixes. A radix tree stores shared beginnings together and keeps track of where sequences diverge. The point for a learner is the saved work: matching prompt prefixes can reuse earlier computation (the SGLang paper).
vLLM and SGLang overlap in their purpose and capabilities. It would be misleading to teach “one is for batching, the other is for caching” as a permanent split. Treat them as alternatives to evaluate for a particular model and workload. Their documentation describes capabilities; it does not establish a universal winner for your service.
Compare the same job
A server comparison asks a narrower question than “Which model is best?” If the model, prompt or number format changes at the same time as the engine, a different answer or speed cannot be attributed to the engine alone. A fair comparison states what stayed fixed and what changed.
- Model and output
- Use the same model revision, tokenizer, chat template, precision and generation settings. A chat template is the model-specific way messages become input tokens.
- Hardware and load
- Use the same devices and input/output-length workload. State how many requests arrive per second and how many can be in progress at once.
- Cache conditions
- Distinguish an empty cache from one containing reusable prompt work. Do not give only one engine the benefit of an already processed prefix.
- Results
- Report initial wait, output pace, total throughput, errors, cutoffs and task quality. Include slow requests, not only the average.
This is a recommended comparison design. The vLLM benchmark tools expose workload controls and timing reports that make these differences visible. Keep software versions and settings with the results so the comparison can be repeated.
Figure 1. A faster-looking result can be incomplete
Illustration Invented results for the same 100 requests, each required to return a complete valid record.
- Setup A
- 100 complete valid records in 20 seconds
- Setup B
- 80 complete valid records and 20 cutoffs in 10 seconds
If the real question is “What is the best system I can deploy?”, each engine may deserve its own tuning. That is a different experiment from isolating engine overhead. Label the scope, keep the hardware budget and workload comparable, and report the configuration for each. Do not present a separately tuned comparison as a one-variable test.
A valid shape can hold a wrong value
JSON is a text format that stores named fields and values. A schema describes the allowed structure, such as a required field named count containing an integer. Constrained generation restricts what the model can emit to follow a supported schema or grammar. SGLang documents these output constraints.
The distinction is easiest to see with a made-up task. The input says “There are two red balls.” Both {"count": 2} and {"count": 7} can satisfy a schema that only requires an integer count. Only the first has the right value. This is an illustration of what the constraint checks, not a model benchmark.
That gives two separate checks: does the output follow the required structure, and does it answer the task correctly? Neither check replaces the other. Also handle incomplete or failed requests: an output cut off mid-record is not a successful structured response. The evaluation section of the fine-tuning piece shows how to report failures and denominators honestly.
A useful serving setup fits the model, returns answers at an acceptable pace and keeps the task working. An engine's advertised features tell you what to investigate. Measurements on your workload and checks of its answers tell you whether it is ready.
Keep the foundations close