The long read
Will this model fit on my GPU?
A model can fit when you load it and run out of memory when you ask it a question. The missing piece is working space: its learned numbers are only part of what must fit. Start with what a GPU does, estimate the space for the weights, then account for the work around them. You will also see why having enough memory does not guarantee a fast reply.
Key ideas, in six lines
- Inference uses existing weights to produce output; training changes weights.
- Parameter count and bits per weight estimate weight storage, not total running memory.
- Quantization reduces storage by approximating numbers; test quality and speed on the actual setup.
- Cached context and temporary work need memory too, so loading a model is only the first check.
- Training keeps extra information for weight updates and can need much more memory than inference.
- Capacity says how much fits; bandwidth says how quickly data moves. Neither alone predicts reply speed.
New to tokens and how a reply is built? Start with What happens when you talk to a chatbot. Select text to underline it or add a pencil note. Your marks stay in this browser.
Running a model or changing it?
A model contains learned numerical settings called parameters, often called weights. Training adjusts them. Inference means using the existing weights to produce an output, such as an answer to a question. In ordinary text generation, the weights stay fixed while the input and generated tokens change. A token is a piece of text, often a word or part of one (Hugging Face text generation).
The label “8B parameters” means roughly eight billion learned numbers. It does not mean eight billion facts, eight billion words, or a score for answer quality. The count helps estimate how much space the weights need; how those numbers are stored matters too (Hugging Face model memory guide).
Keep the question precise: “Will it fit for inference?” and “Will it fit for training?” have different answers. For how a training run changes the model, follow What happens when you fine-tune a model. This piece first works out the needs of running one.
The CPU, the GPU and their memory
A CPU, the central processing unit, runs the program and coordinates work. A GPU, the graphics processing unit, can carry out many numerical calculations in parallel. That suits much of the repeated maths in a model. The CPU still has a job: using a GPU does not replace the rest of the computer (NVIDIA programming model).
A discrete GPU usually has its own memory, commonly called VRAM, separate from the computer's main RAM. A machine with 64 GB of system RAM and a 16 GB GPU does not automatically have 80 GB of GPU memory. Moving data between the pools takes work. Unified-memory machines share memory differently, so this distinction depends on the hardware (NVIDIA host and device memory).
Read the memory capacity for the device that will run the model. Then check what is actually free: another program or model may already be using part of it. “Out of memory”, often shortened to OOM, means a needed allocation could not be satisfied. It does not, by itself, tell you which part of the workload caused the shortage (PyTorch memory management).
Count the space for the weights
A bit is one binary digit. Eight bits make one byte. A 16-bit number therefore takes two bytes. FP16 and BF16 are two formats for representing numbers in 16 bits. They use the same storage per number but have different ranges and precision: range is how large or small a value can be, while precision is how finely nearby values can be distinguished (PyTorch data types).
For a simple weights-only estimate, multiply the parameter count by bytes per weight. Suppose an 8 billion parameter model stores every weight in 16 bits. The arithmetic is:
| Quantity | Value |
|---|---|
| Parameters | 8 billion |
| Bytes per weight | 16 bits ÷ 8 = 2 bytes |
| Weight storage | 8 billion × 2 = 16 billion bytes |
| In decimal GB | About 16 GB, weights only |
That is a calculation, not a measured hardware requirement. It does not show that a 16 GB GPU can run the model. The working space still has to fit. Decimal GB means a billion bytes; GiB means 1,073,741,824 bytes. When two tools display different memory numbers, check their units before comparing them (NIST binary prefixes).
Going deeper: stored precision and calculation precision
A program can mix number formats. Some values may use 16 bits while particular calculations or stored copies use 32 bits. This is mixed precision. A single precision label therefore does not describe every buffer in memory or every operation. Supported formats also depend on hardware and software (NVIDIA mixed precision guide).
Leave room for the work
Loading the weights is one stage. Producing an answer adds temporary calculations and other working data. Many text-generation systems also keep a cache of calculations for earlier tokens. These share the available memory, along with the runtime, the software executing the model (Hugging Face inference memory guide).
Keep these parts separate when asking what will fit:
| Part | What it holds |
|---|---|
| Weights | The model's learned numbers |
| Cached context | Reusable calculations for earlier tokens |
| Working space | Intermediate results and temporary calculations |
| Runtime overhead | Memory used by the execution software |
This is a list of components, not a fixed percentage split. A short request and a long request can need different working memory on the same model. A useful fit check therefore includes the prompt length, output limit and number of simultaneous requests you intend to support, not just a successful model load (Hugging Face inference memory guide).
What fewer bits buy you
Quantization represents numbers with fewer bits, approximating the original values. Fewer bits per weight means less weight storage. But the approximation can change answers, and the speed depends on how well the hardware and software handle that format. Smaller is not a promise of faster or equally accurate (Hugging Face quantization concepts).
In the same weights-only arithmetic, eight billion 4-bit weights occupy 4 GB. That is one quarter of the 16-bit estimate. A real quantized model also carries information needed to interpret those compressed values, and some components may remain at higher precision. Its cache and working space do not automatically shrink by the same factor (quantization formats and metadata; bitsandbytes implementation).
When you hear that QLoRA uses a 4-bit base model, this is the storage idea behind it. QLoRA trains small additions while keeping that compressed base fixed. It does not mean every training value uses four bits. For its role in training, return to LoRA and QLoRA (Hugging Face).
Why a longer chat can need more memory
Attention is the part of a transformer model that uses information from other tokens. A key-value cache, usually called the KV cache, saves some attention calculations for tokens already processed. The next generation step can reuse them instead of recomputing them. This saves work by spending memory (Hugging Face caching explanation).
The cache is working data for generation. It is not a change to the learned weights, and it is not the same as the chat history an app saves for you. Its contents can be discarded and recomputed from the tokens (how cached calculations are reused).
With ordinary full attention, keeping more tokens in the cache takes more space. More simultaneous requests can also require more cache. So a model that handles one short prompt may run out of memory on a long prompt or several requests together. Some architectures limit cache growth, and implementations may share or compress parts of it. There is no universal number of bytes per token for every model (Hugging Face cache strategies).
Suppose a short chat works but a much longer chat fails on the same loaded model. The weights did not grow. The longer request is evidence to investigate cache and working space; it does not prove which allocation failed. Change one setting at a time and inspect memory use before choosing a remedy.
Why training needs extra space
Training needs information that ordinary inference does not keep. Activations are intermediate results, often saved for the backward pass that works out how to change the weights. Gradients indicate how a change to a weight affects the training loss. The optimizer, the update algorithm, can keep additional state of its own. Those stores can make training require much more memory than running the same weights (Hugging Face training memory anatomy).
A larger training batch processes more examples together and usually needs more space for intermediate results. Longer examples can also increase memory use. There is no single multiplier that converts inference memory into training memory: the method, optimizer, formats and workload matter (training memory components).
Going deeper: gradient accumulation
Gradient accumulation combines gradients from several small batches before updating the weights. Suppose one GPU processes 2 examples at a time and accumulates 4 such batches. The effective batch is 8 examples per update, without holding all 8 examples' intermediate results at once. This changes the amount of work per update; it is not a guarantee of faster training or better quality (Hugging Face Trainer).
Accumulation does not remove the memory needed for the weights, gradients or optimizer state. It helps when the simultaneous batch is the part that is too large. Another technique, gradient checkpointing, saves fewer intermediate results and recomputes some during the backward pass, trading extra work for memory (Trainer memory settings).
Enough space is not the same as speed
Memory capacity says how much fits. Memory bandwidth says how quickly data can move. These answer different questions:
| Measure | How to read it |
|---|---|
| Capacity: GB | More means more space, not necessarily faster replies. |
| Bandwidth: GB/s | More means more data can move per second, not that every task gets faster. |
If an operation waits mainly for data, it is memory-bound. If it waits mainly for arithmetic, it is compute-bound. Improving the part that is not limiting the work may do little. The limiting part can change with the workload, so “this GPU is faster” needs a task and test conditions (NVIDIA performance guide).
Going deeper: what TFLOPS does and does not measure
TFLOPS means trillions of floating-point operations per second. A spec sheet usually gives a peak arithmetic rate for a particular format and set of conditions. Twice the peak TFLOPS does not promise twice the generated tokens per second: data movement, the model, request sizes and execution overhead still matter (NVIDIA GPU performance background).
Compare like with like: the same model and precision, input and output lengths, and request load. Measure the result people experience. The inference speed guide separates the wait before output starts from the speed of the continuing reply.
Where CUDA and kernels fit
CUDA is NVIDIA's platform for programming GPUs. A kernel is a function launched on the GPU to perform work in parallel. These are parts of the execution machinery; they are not trained models (NVIDIA programming model).
A serving engine is the software that loads models, accepts requests and manages their execution. Its work may use GPU kernels underneath. That gives the names a place: model weights supply the learned numbers, the engine manages requests, and kernels perform GPU operations. Continue with the serving guide, including vLLM and SGLang (vLLM documentation).
Sources and further reading
The examples are arithmetic and diagnostic exercises, not measured GPU benchmarks. Model support and runtime behaviour change; use the documentation for the version you run.
- Hugging Face: text generation.
- Hugging Face: optimizing language models for speed and memory.
- NVIDIA: the CUDA programming model.
- PyTorch: CUDA memory management.
- PyTorch: tensor data types.
- NIST: prefixes for binary multiples.
- NVIDIA: mixed precision training.
- Hugging Face: quantization concepts and bitsandbytes.
- Hugging Face: how caching works and cache strategies.
- Hugging Face: training memory anatomy and Trainer settings.
- NVIDIA: deep learning performance and GPU performance background.
- vLLM documentation.
It fits. Why is the reply slow?