AI Dojo Back to the typewriterThe typewriter

The long read

What happens when you fine-tune a model

22 min read
Assumes: you know how a chatbot produces a reply.
First published 29 October 2026. Last edited 16 November 2026.

A model you can download or rent already writes fluent text. Fine-tuning trains it a little further on your own examples, so a behaviour you want becomes part of the model instead of something you ask for every time. This piece follows one fine-tune from the first question, whether to do it at all, to keeping it working once people use it. Plain language comes first, with deeper notes along the way for practitioners.

Key ideas, in nine lines
  • Fine-tuning changes the weights, the model's learned numerical settings. Prompts, retrieval and tools leave them unchanged.
  • Measure prompting, retrieval or tools first as appropriate. Consider fine-tuning for behaviour, format, cost or speed that still misses your goal.
  • For a changing knowledge base, try retrieval first: it supplies current source passages. Fine-tuning can learn some facts, but keeping them current and traceable is harder.
  • Training reinforces patterns in examples, including mistakes. Cover the task's different cases; adding more of the same is not enough.
  • Lower loss means a better fit to the scored tokens under the same setup. Judge the run on unseen examples and the intended task too.
  • LoRA trains a small correction beside unchanged base weights. QLoRA also compresses the base weights. Both save training memory; test the resulting quality.
  • Compare with a strong baseline on held-out tasks. Name what every rate measures, report failures and timeouts, and re-check safety.
  • Preference tuning and reward-based training teach by comparison or score, and every reward can be gamed.
  • A shipped fine-tune needs its licence checked, monitoring, and retraining when the evidence says so.

Select any line to underline it in pencil or to write a note beside it. Your marks stay in this browser. Every concept named here, and many more, is in Fine-tuning, every concept.

Should you fine-tune at all?

A language model's behaviour is set by a very large set of numbers called , or weights. Training is the process of setting them. means taking a model that has already been trained and training it a bit more on your own examples, so those numbers change and the new behaviour is built in. The result is a new model, sometimes called a custom or tuned model, that you have to store, test and look after.

Most other ways to change what a model does leave the weights alone. A is the text you send; a is a standing instruction placed first. Few-shot examples are worked examples inside the prompt that help a model follow a pattern without updating its weights (Brown et al. 2020). searches your documents when a question arrives and puts relevant passages into the prompt (Lewis et al. 2020). Tool use lets the model ask your software to run a function, such as a database lookup, and use the result. These methods change what the model receives, which can change its answer without retraining it.

The vendors who sell fine-tuning tell you to try those first. Google's tuning guide says to start with prompting and move to fine-tuning only if required (Google Cloud); Mistral's says prompting is faster and less resource-intensive (Mistral); OpenAI's recommended workflow begins by writing evaluations to set a baseline (OpenAI). That gives a sensible order, shown in Figure 1.

Figure 1. Try these in order

A default order drawn from the vendor guidance above. Each step is cheaper to try and easier to undo than the next.

  1. Prompt wellClear instructions, a system prompt, a few examples. Measure the result on a test set.
  2. Still missing knowledge?
  3. Add retrievalFetch the right documents for each question, so facts stay current and citable.
  4. Needs live data or actions?
  5. Add toolsLet the model call your systems.
  6. Behaviour, format, cost or speed still wrong?
  7. Fine-tuneTrain on examples of the behaviour, then test again, including tasks you did not train on and safety.

Fine-tuning can make behaviour more consistent: a fixed output format, a house tone, one classification scheme, a narrow repeatable task. It can also teach a smaller, cheaper, faster model to do a job a bigger one was doing. None of those gains is automatic; measure the behaviour you want. OpenAI lists shorter prompts, training on sensitive data once instead of sending it with every request, and moving to a smaller model as reasons to fine-tune (OpenAI).

For answers from a changing knowledge base, retrieval is a sensible first choice. Ovadia et al. compared retrieval with further training on raw documents across knowledge-intensive tasks. Retrieval did better on their tested tasks, for both existing and new knowledge; this does not establish that every form of fine-tuning loses on every task (Ovadia et al. 2023). Another controlled study found that unfamiliar facts in training examples were learned more slowly and could increase , confident false answers, in the models it tested (Gekhman et al. 2024). Fine-tuning can learn some facts, as the deeper notes discuss, but facts stored in weights can become stale and are harder to trace to sources.

Some other things fine-tuning cannot do:

  • Keep a model up to date, because the weights freeze when training ends.
  • Reach live data.
  • Show where a fact stored in the weights came from: the weights keep no record of their sources. A model can be trained to cite passages placed in its prompt or found by retrieval, as Self-RAG does, with better citation accuracy reported for long answers (Asai et al. 2023); the citations then point to those passages, not to the weights.
  • Rescue a task nobody has defined well enough to write good examples for.

The order in Figure 1 is a default, not a law. Fine-tuning can be the goal from day one when you need a small, cheap model at high volume, when the examples you need no longer fit in a prompt, or when you start from a base model, which continues text but does not yet follow instructions well.

Going deeper

"Fine-tuning" is an umbrella for several methods, told apart by the data they train on.

Continued pretraining
Raw text from one field, such as law or medicine, still predicting the next word (Gururangan et al. 2020).
Supervised fine-tuning
Input and ideal-output pairs.
Preference tuning
Pairs of better and worse answers.
Reinforcement fine-tuning
Scores from a grader.
Distillation
A larger model's answers.

Base and instruct models are stages of one pipeline: in the InstructGPT study, labelers preferred outputs from a 1.3 billion parameter InstructGPT model over outputs from the 175 billion parameter GPT-3, despite it having about 100 times fewer parameters (Ouyang et al. 2022).

Contested: can fine-tuning teach facts at all? Partly, say some studies: facts had to be augmented, for example paraphrased many ways, before a model could extract them (Allen-Zhu and Li 2023), and training data generated to cover each fact evenly improved uptake (Mecklenburg et al. 2024). The disagreement is about how much data engineering it takes and whether that beats retrieval. Even when it works, facts in the weights go stale and cannot be traced to a source the way retrieved passages can.

What a training example looks like

The most common kind is . You collect examples, each with an input and a target response showing the behaviour you want. Training encourages the patterns in those responses, including tone, length and format. It can also reinforce mistakes. If every example is short, that encourages short answers, whether you meant it to or not. This is learning from examples; it need not mean copying each answer word for word (TRL docs).

Chat data is a list of messages, each tagged with a role: "system" for standing instructions, "user" for the person, "assistant" for the model, and "tool" for the result of a function the model asked to run. The usual file format is JSONL (JSON Lines), a text file in which every line is one complete training example (OpenAI). Figure 2 shows two lines.

Figure 2. Two training examples in JSONL

Illustration not real data. The first line is one turn; the second is a short conversation.

{"messages": [{"role": "system", "content": "Answer in one sentence."}, {"role": "user", "content": "What is a lease?"}, {"role": "assistant", "content": "A lease is a contract to rent property for a set time."}]}{"messages": [{"role": "user", "content": "Convert 5 km to miles."}, {"role": "assistant", "content": "About 3.1 miles."}, {"role": "user", "content": "And 10 km?"}, {"role": "assistant", "content": "About 6.2 miles."}]}
One line, one example (numbered here for reading; the file has no numbers). A single malformed line can fail an upload.

A model never reads that file directly. A turns the list of messages into one long string, with special markers, called special tokens, that say who is speaking and where each turn ends. Each chat model ships with its own template, and two models built from the same base can use different ones. Hugging Face's documentation warns that with the wrong control tokens these models perform drastically worse (Hugging Face). The end-of-turn marker matters more than it looks: it is how the model learns to stop.

A token is a piece of text the model reads or predicts. Training scores how well it predicts the example's next tokens using a penalty called loss. Lower loss means a better fit to those tokens when the data and scoring setup are the same. It is not a percentage, and it does not check whether an answer is true or useful. The training section works through the numbers; evaluation checks the task itself (TRL docs).

chooses which tokens count toward that penalty. In assistant-only training, the reply and its end-of-turn marker count, while the prompt does not. The model still reads the prompt as context. Scoring the marker teaches the model when the answer should end. Other setups also score the prompt; assistant-only scoring is a choice for the task, not a universal rule (TRL docs).

Figure 3. Which tokens are scored

Illustration the first example from Figure 2, after the chat template. Real templates add their own markers.

  1. systemAnswer in one sentence.read, not scored
  2. userWhat is a lease?read, not scored
  3. assistantA lease is a contract to rent property for a set time. [end of turn]scored
The end-of-turn marker is scored too, which is how the model learns where to stop.

A training run sets its own maximum length for each example, in tokens. That configured limit can be shorter than the model's own context maximum. The prompt, the answer and the chat-template tokens have to fit inside the configured limit together. The template's markers say who is speaking, and they count even though you did not write them as ordinary words. Past the limit, shorten, split or filter the example. With truncation on, the extra tokens are cut; some pipelines reject the example instead. Illustration, with numbers chosen so the sum is exact, not measured from a real template: a configured limit of 1,000 tokens, a prompt of 700, an answer of 250 and 50 template tokens. 700 + 250 + 50 = 1,000, so that example just fits. One more token and it no longer fits. It is cut only if truncation is on.

Going deeper

A template mismatch is a silent bug: training runs, the loss falls, and the deployed model still misbehaves. Hugging Face documents three causes: the wrong control tokens; special tokens added twice, when text rendered by the template is tokenized again with special tokens switched on; and a missing generation prompt at inference, which can make the model continue the user's message (Hugging Face). A misaligned end-of-sequence token stops replies ending cleanly (TRL docs). The cheapest check is to decode one fully prepared example back to text and read it, along with which tokens are scored.

In the TRL library, with truncation, the default maximum length is 1,024 tokens and the cut keeps the start, so the end of a long example, often the answer, is what gets dropped (TRL docs, version 1.16, as of 28 October 2026). That is this tool's truncation, not what every pipeline does, and the configured length can be shorter than the model's own maximum. It is separate from the 1,000 token illustration above. Measure the prepared example, template tokens included, before you train.

In TRL, prompt and completion data uses completion-only loss by default, and conversational data needs assistant_only_loss=True plus a template that marks assistant text (TRL docs, version 1.16.0, as of 28 October 2026). Contested: masking is a sensible default, not a law. Also scoring the instruction helped across 21 benchmarks in one study, most when instructions were long, outputs short and examples few (Shi et al. 2024).

The data decides what it learns

Correctness and coverage matter as much as count. A small clean set can beat a larger noisy one, but harder or broader tasks may still need many varied examples. LIMA fine-tuned a 65 billion parameter model on 1,000 curated examples; in its human study the answers matched or beat GPT-4's in 43% of cases. Doubling data without adding variety did not improve that study's results (Zhou et al. 2023). The QLoRA authors also found a 9,000 example dataset beat a 450,000 example one on their chatbot evaluation (Dettmers et al. 2023). These are reasons to examine what examples teach, not a universal target dataset size.

Start with enough careful examples to run a useful small test. If it shows no gain, check the task, prompt, example quality and evaluation before adding more. Does the test measure the behaviour you meant to teach? Then collect examples for the failures you observe, rather than raising the count without a reason. Provider starting counts are only starting points, as the reference note records.

Provider starting counts: reference note

Checked 28 October 2026: OpenAI's guide accepted a minimum of 10 examples and suggested starting with 50 well-crafted ones; if 50 had no effect, it advised rethinking the task or prompt before adding data (OpenAI). Google suggested starting with 100 (Google Cloud). Check the current guide for your model before a run; these dated counts are not guarantees or minimum learning requirements.

Examples come from three places: people who know the task (slow, but the most control), curated logs of real conversations (realistic, but full of mistakes and personal data), and synthetic data written by a model (fast, but it inherits that model's blind spots). A common version is , where a large "teacher" model writes answers and a smaller "student" is trained on them. Students trained to imitate a strong chat model copied its style well but closed little to none of the gap on tasks the imitation data did not cover (Gudibande et al. 2023). Check the teacher's terms too: OpenAI's terms of use forbid using output to develop models that compete with OpenAI (OpenAI terms).

Cleaning matters as much as collecting. Remove duplicates, which over-weight whatever they repeat and make memorisation more likely (Lee et al. 2021). Filter out examples that are wrong, off-topic, cut off or unsafe. Keep labels and format consistent, or the model learns a blur of both styles. Cover every kind of request the model will meet, including awkward edge cases, polite refusals, and requests that look risky but are fine, so it does not learn to refuse safe ones. Record where each example came from, so a bad batch can be found and removed later.

Then split the data three ways.

Training set
What the model learns from.
Validation set
What you look at while making choices, such as which settings to keep.
Test set
Kept aside for the final check. Also called the .

Scoring a model on data it trained on is, in scikit-learn's words, "a methodological mistake", and choosing settings by looking at the test score leaks information too (scikit-learn). Keep related examples together: if one turn of a conversation is in training and the next is in the test set, the model can score well by recognising details.

A checkpoint is a snapshot saved during training. Once you keep the checkpoint with the best validation score, that score is no longer an unbiased final measure: you chose the one that looked best there. The test set, which took no part in the choice, is the final check.

Logs and transcripts are full of personal data. Training on it needs a lawful reason, and models can memorise and repeat training text: researchers extracted names, phone numbers and email addresses verbatim from GPT-2 (Carlini et al. 2020). In the EU, the European Data Protection Board says a model built with unlawfully processed personal data may make its later use unlawful too (EDPB). Remove what the task does not need. This is not legal advice.

Going deeper

Contested: how much data is enough? LIMA's "Superficial Alignment Hypothesis" holds that knowledge and skills come almost entirely from pretraining, and fine-tuning mostly teaches format (Zhou et al. 2023). A later study found performance on maths, coding and multi-step reasoning keeps rising with more examples, and called the hypothesis "at best, an over-simplification" (Raghavendra et al. 2024). Small sets work well for tone and format; harder skills may need more.

Synthetic data scales fast: Alpaca built 52,000 instruction examples from 175 human-written seeds for under $500 of API cost (Stanford Alpaca). Training models over and over on model-made data can make rare patterns fade, called model collapse (Shumailov et al. 2023); keeping the real data alongside avoided it in one set of experiments (Gerstgrasser et al. 2024). Before training, also search your data for any public benchmark questions you plan to report on, because contamination inflates scores (Sainz et al. 2023).

What happens during training

Training repeats one loop.

  1. Show the model a batch of examples.
  2. At each position it predicts the next token.
  3. Score the predictions against the example's target tokens using a penalty called loss.
  4. Work out, for every weight being trained, which direction would lower the loss (the gradient, computed by backpropagation).
  5. Let the optimizer, usually one called AdamW, nudge each weight a little.

One update is a step. One full pass over the training data is an . More epochs mean revisiting the same examples. That raises the risk of overfitting, fitting those examples so closely that new inputs get worse, and of forgetting earlier skills. More passes do not guarantee better answers on new inputs.

Lower loss means the model fitted the scored example tokens better. Compare losses only when the data and scoring setup are the same: changing the tokenizer, context length or which tokens are scored changes the comparison (Hugging Face). In supervised language-model training, the usual loss is cross-entropy: take minus the natural logarithm of the probability assigned to each target token, then average over scored tokens (TRL docs). The target token is the one supplied by the example, not an independent truth label. Illustration:

Chance assigned to the example's target tokenLoss: lower is better for this score
90%0.105
50%0.693
1%4.605

A loss is not a percentage: 0.16 does not mean 16% wrong. In this illustration, raising the target token's chance from 50% to 90% lowers loss from about 0.69 to 0.11. Halving its probability adds about 0.693 to loss, whether it falls from 0.6 to 0.3 or from 0.1 to 0.05. A near-zero training loss shows a close fit to the examples, but does not establish useful answers to new questions. Check held-out examples and your intended task.

The controls how large each weight update is. Too small can make learning slow; too large can make training unstable or damage useful behaviour. Choose it using validation results. Defaults differ by tool and training method: one study found LoRA's best learning rate about ten times higher than full fine-tuning's on its tested tasks (Biderman et al. 2024).

Two counts are easy to mix up. The microbatch is how many examples one device runs in one forward and backward pass. The effective batch is how many examples sit behind one weight update. When the batch you want will not fit in memory, gradient accumulation runs several smaller microbatches and combines their gradients, the directions that would lower the loss, and normalizes them, then updates the weights once. Without that normalization, the update would grow with the number of passes. Illustration, on one GPU: microbatch 1 with 8 accumulation steps is an effective batch of 8. Under the same loss normalization, that update is comparable to a true batch of 8, not guaranteed identical. That saves memory for intermediate results, not for the weights or the optimizer state.

Watch two curves. Training loss is measured on examples used to update the weights. Validation loss is measured on held-out examples. If training loss falls while validation loss rises, the model fits training examples better but predicts held-out examples worse. That is an warning: fitting the training set too closely can hurt performance on new inputs (Google). Persistently poor results on both sets can indicate underfitting. Early stopping ends a run when a chosen validation measure stops improving. Save checkpoints, snapshots during training, and compare them on the intended task; the last one need not be best (Microsoft).

Figure 4. What overfitting looks like

Illustration the shape, not real training data. Solid line: training loss. Dashed line: validation loss.

LossTraining stepsWarningTrainingValidation
Training fits better while held-out prediction gets worse. A lower training loss alone does not prove better answers; judge task results too.

Suppose training loss falls, but answers to new questions get worse. That run has not succeeded on the task, even though its training score improved. Compare the saved checkpoints on the same held-out tasks and inspect the regressions before choosing a model. Loss and task results measure different things; the deeper notes give an example where they moved in different directions.

Training on something new can wear away what the model knew before. This is , and it was observed generally in models from 1 to 7 billion parameters during continued fine-tuning (Luo et al. 2023). Forgetting tends to grow the longer you train (Biderman et al. 2024). Defences include mixing some general data back in, called replay: one model kept almost all its earlier performance while learning 8 new tasks with a 1% rehearsal buffer (Scialom et al. 2022). Fewer epochs and LoRA, described next, also help.

Going deeper

Illustration, counting steps: a microbatch of 2 per GPU, 8 accumulation steps and 4 GPUs give an effective batch of 64 examples per weight update. With 6,400 examples that is 100 steps per epoch and 300 steps for 3 epochs, and a 5% warmup, which starts the learning rate near zero and ramps it up, lasts 15 steps. As of 28 October 2026, Transformers (version 5.21.0) defaults to 3 epochs, and Unsloth's guide suggests 1 to 3 (Transformers, Unsloth).

Teacher forcing and exposure bias. In training, every prediction is made from the true previous tokens of the example, never from the model's own earlier guesses. This is called teacher forcing (Goodfellow et al. 2016). When the model generates, it reads its own earlier output instead, so an early mistake can carry forward and errors can accumulate. Bengio et al. described this discrepancy and proposed scheduled sampling, gradually feeding the model its own predictions during training (Bengio et al. 2015); Ranzato et al. named it exposure bias (Ranzato et al. 2016). Both papers studied recurrent networks, but language models are still trained this way: in TRL, the labels are the input shifted by one position, so each position learns to predict the true next token (TRL docs).

Loss is not the task. InstructGPT's validation loss overfit after 1 epoch, yet training for more epochs still improved reward model scores and human preference (Ouyang et al. 2022). Pick checkpoints by task evaluations, not loss alone. Perplexity, the exponential of the average loss, carries the same information on another scale: a loss of 2.0 is a perplexity of 7.39. It depends on the tokenizer, so it cannot be compared across models with different tokenizers (Hugging Face).

Starting learning rates as of 28 October 2026: Hugging Face Transformers (version 5.21.0) uses 5e-5 (0.00005), TRL (version 1.16.0) uses 2e-5 for full fine-tuning and suggests about 1e-4 for adapters, and Unsloth's guide suggests 2e-4 for LoRA (TRL, Unsloth). Other Transformers 5.21.0 defaults on the same date: gradient clipping at a norm of 1.0, a linear learning-rate schedule, weight decay 0 and seed 42 (Transformers). Defaults drift between versions, and PyTorch says fully reproducible results are not guaranteed across releases or platforms (PyTorch), so record library versions, the seed, the data version and the base model revision with every run. Lowering the learning rate to reduce forgetting is common practice, but no single controlled study we found isolates it for language models.

Full fine-tuning, LoRA and QLoRA

Full fine-tuning updates every weight. Training needs memory not just for those weights but also for gradients, the directions of change, the optimizer's working records, and intermediate calculations. A GPU is a processor that performs many of these calculations in parallel; its memory must hold the working data. The GPU and memory guide explains those pieces. Hugging Face's example for mixed-precision training with the Adam optimizer counts 6 bytes per parameter for weights, 8 for the optimizer and 4 for gradients, before intermediate results (Hugging Face). Illustration: 18 bytes times 7 billion parameters is about 126 GB before that extra working memory. Requirements depend on the training setup.

Parameter-efficient fine-tuning (PEFT) saves training memory by updating a small share of the parameters and leaving the rest frozen, meaning unchanged (Hugging Face PEFT). , low-rank adaptation, trains a small set of extra numbers called an adapter. The adapter supplies a correction to the original model. LoRA represents that correction using two small matrices, tables of numbers, beside each chosen original matrix. Training fewer numbers reduces the memory needed for gradients and optimizer records. The original weights still need to be stored.

After training, the correction can be added into the original weights so there is no separate adapter calculation when the model runs. On GPT-3 175B, the LoRA paper reports about 10,000 times fewer trainable parameters, about a third of the GPU memory (1.2 TB down to 350 GB) and a saved checkpoint shrinking from 350 GB to 35 MB (Hu et al. 2021). Those are results from that setup, not a memory estimate for every model. Another PEFT method, BitFit, updates only existing bias terms (Ben-Zaken et al. 2021).

Figure 5. What LoRA adds to one weight matrix

Illustration one 4,096 by 4,096 matrix with a rank 16 adapter.

Frozen matrix
4,096 × 4,096 = 16,777,216 parameters
Adapter, two thin matrices
2 × 16 × 4,096 = 131,072 trainable parameters
Share trained
131,072 ÷ 16,777,216 ≈ 0.78%
The rank (16 here) is the shared width of the thin matrices: how many independent directions the correction can use. A higher rank trains more parameters. It does not set how large the change is.

That rank is the shared width of the two thin matrices: how many independent directions the correction can use. It does not set how large the change is. A rank of 1 can still be a large change. The size is set separately, including by alpha, a scaling setting described in the deeper notes below. A higher rank trains more numbers and uses more memory. It does not by itself mean better answers. The LoRA authors found that a rank as small as one or two was enough for GPT-3 175B on the tasks they tested, and that raising the rank did not always help (Hu et al. 2021). Rank is not a count of layers, and it is not a count of training examples. Which matrices receive an adapter is a separate setting.

saves more memory by storing the frozen base model in 4-bit form while training a LoRA adapter. Bit width is the space used to represent each number; a bit holds a 0 or a 1, and eight bits make a byte. represents weights with fewer bits. The adapter and training calculations use higher precision, so the whole process is not simply 4-bit. Its authors fine-tuned a 65 billion parameter model on one GPU with 48 GB of memory while matching 16-bit fine-tuning on their tested tasks (Dettmers et al. 2023). That is an attributed experiment, not a promise that any 65-billion-parameter model or training job fits.

Training a LoRA adapter in 16-bit, merging it into the base, then quantizing the merged model to 8 bits for serving is not QLoRA. QLoRA trains against a base stored in 4-bit form. Quantizing after the merge is inference compression: it changes the file you serve, not how the base was stored during training.

Illustration: 7 billion weights at 4 bits (half a byte each) use about 3.5 GB for the weight values, compared with 14 GB at 16 bits. Both estimates exclude adapters, quantization metadata, optimizer records and working memory. The memory guide shows why the weight file size is only the beginning.

Contested: does LoRA match full fine-tuning? The LoRA and QLoRA papers report matching it on their benchmarks. A later study found LoRA clearly behind full fine-tuning on code and maths, while forgetting less of the base model's other skills, and summed it up as "LoRA learns less and forgets less" (Biderman et al. 2024). Which matters more depends on how far your task is from what the base model already does.

Going deeper

LoRA has three main settings.

Rank
The shared width of the two thin matrices: how many independent directions the correction can use. It does not set the size of the change, and it is not a count of layers or of examples. As of 28 October 2026, the PEFT library (version 0.21.2) defaults to 8, and Unsloth's guide suggests 16 or 32 (PEFT, Unsloth).
Alpha
Scales the size of the correction by alpha divided by rank. Rank itself is not that size.
Target modules
Which matrices get adapters; the QLoRA authors found that adapting all the linear layers was needed to match full fine-tuning (Dettmers et al. 2023).

LoRA learns B (d by r) and A (r by k) beside a frozen W, computing W x + B A x; B starts at zero, so training begins from the unchanged model (Hu et al. 2021). rsLoRA scales by alpha over the square root of rank instead, so r 16 and alpha 32 give a scale of 8 rather than 2, which keeps high-rank adapters learning (Kalajdzievski 2023). Biderman et al. recommend alpha of twice the rank, and rank 256 for code if memory allows. DoRA splits each weight into a size and a direction and uses LoRA for the direction (Liu et al. 2024). Older bottleneck adapters add small layers inside the model, which adds latency at inference (Houlsby et al. 2019).

Memory tricks each shrink a different bucket: 16-bit mixed precision (Micikevicius et al. 2017), gradient checkpointing, which recomputes intermediate results instead of storing them (Chen et al. 2016), 8-bit optimizers at 2 bytes per parameter instead of 8 (Dettmers et al. 2021), and sharding weights, gradients and optimizer states across GPUs with DeepSpeed ZeRO or PyTorch FSDP (DeepSpeed). Common open tools as of 28 October 2026 are Transformers, TRL, PEFT, Axolotl, Unsloth and LLaMA-Factory; torchtune's README says it is no longer actively maintained (torchtune).

Did it work?

Decide what "better" means before you train, in terms a stranger could score. Anthropic's documentation says building with a model starts with clearly defining success criteria and designing evaluations to measure them (Anthropic). OpenAI's fine-tuning guide puts it more bluntly: "Good evals first!" (OpenAI).

Compare against honest baselines on the same test items, with the same scorer: the original model with a carefully written prompt, the model exactly as it was before your run, and often a stronger model as a reference. Comparing your fine-tune with a lazily prompted base model inflates the gain, because part of it could have come from a better prompt alone. Score both models on identical items and read the list of items that flipped from pass to fail, which is where regressions hide (Miller 2024).

Use the cheapest check that works. Deterministic checks, small programs that give the same verdict every time, can test whether output is valid JSON, follows a schema, uses an allowed label or fits a length limit. Many fine-tunes exist mainly to pass checks like these. Exact match suits answers with one right value. Overlap scores such as BLEU and ROUGE compare text with a reference answer, but measure similarity, not correctness.

Two settings at generation time can quietly spoil a test. The cap on new tokens, called max_new_tokens in Transformers, is separate from the length of the prompt, and as of 29 October 2026 the docs say generate() returns up to 20 tokens by default unless the model's own configuration says otherwise (Transformers). An answer that hits the cap is cut off before its end-of-sequence token, so count cut-off answers as failures or report them separately. Also read answers the model writes freely: decoding by likelihood can produce text that is "bland and strangely repetitive" (Holtzman et al. 2020), and the loss, measured on reference text, will not show it. Check generations for loops and repeated lines.

For qualities a rule cannot check, such as helpfulness or tone, you can ask a capable model to grade, called . A strong judge agreed with human preferences over 80% of the time in one study, about as often as people agree with each other (Zheng et al. 2023). But judges have biases. They can favour whichever answer is shown first: swapping the order let one model beat ChatGPT on 66 of 80 queries, with ChatGPT judging (Wang et al. 2023). They can favour longer answers (Dubois et al. 2024), and their own writing (Panickssery et al. 2024). Judge each pair in both orders, control for length, use a judge from a different model family, and check the judge against people on a sample.

Report every rate with what it measures and what it is out of. Illustration: 200 test items, of which 40 ask for a JSON answer. Of those 40 requests, 36 return an answer that passes the schema check, 3 return an answer that fails it, and 1 times out.

  • 200 test items
    • 160 need no JSON: not applicable
    • 40 need JSON
      • 1 timed out
      • 39 completed
        • 36 pass
        • 3 fail
  • Across all eligible requests: 36 of 40 succeeded, or 90%, counting the timeout as a failure.
  • Among completed evaluations: 36 of 39 passed, or about 92%, with 1 timeout reported separately.

Both rates are valid because they answer different questions. Neither includes the 160 items that did not require JSON. Calling the timeout "not run" would hide what happened: the request was attempted but did not finish in time. Small test sets also carry wide uncertainty. Illustration, using the Wilson interval recommended for small samples (Brown, Cai and DasGupta 2001):

CorrectRate95% interval
18 of 2090%about 70% to 97%
180 of 20090%about 85% to 93%

A two point gap on a few hundred items can be noise, and two training runs that differ only in the random seed can score noticeably differently (Dodge et al. 2020).

Finally, check what might have got worse. Run a fixed set of general tasks before and after, to catch forgetting. Re-test safety every time: one study weakened GPT-3.5 Turbo's guardrails with 10 adversarial training examples at a cost under $0.20. Ordinary datasets also weakened safety in the tested models, to a lesser degree (Qi et al. 2023). This is a risk to check, not a prediction for every run. Test refusals of harmful requests, and also over-refusal of safe requests that only look risky (Röttger et al. 2023). Red-teaming, attacking your own model on purpose, can find failures before users do.

Going deeper

Miller gives the standard error of a mean score as the sample standard deviation over the square root of n, a 95% interval of plus or minus 1.96 standard errors, and recommends computing the standard error on per-item differences for paired comparisons. When items come in groups, such as several questions about one document, clustered standard errors can be about three times larger than naive ones (Miller 2024). The simple normal interval behaves badly on small samples; for 18 of 20 it runs past 100%. How many test items are enough depends on the difference you need to detect, which a power calculation answers; Miller's worked example suggests about 1,000 questions for new evaluations.

Contamination, test items or close paraphrases in the training data, inflates scores, and paraphrased or translated items slip past simple duplicate filters (Yang et al. 2023). Goodhart's law applies too: optimise hard for a proxy score and it can stop tracking what you wanted (Gao et al. 2022). As of 28 October 2026, OpenAI's guide says it assesses fine-tuned models across 13 safety categories and can block deployment, while still asking customers to run their own evaluations (OpenAI).

Beyond examples: preferences and rewards

For many tasks, writing the ideal answer is hard, but telling which of two answers is better takes seconds. Preference tuning uses that easier signal. A preference record holds a prompt, a "chosen" answer and a "rejected" one.

Figure 6. One preference record

Illustration not real data.

{"prompt": "Explain a mortgage to a 12-year-old.",
 "chosen": "It's a big loan to buy a house. You pay it back a bit each month...",
 "rejected": "A mortgage is a secured debt instrument collateralized by real property..."}

In the classic setup for reinforcement learning from human feedback (), training happens in three stages.

  1. Supervised fine-tuning on demonstrations.
  2. People rank several answers per prompt, and a separate reward model learns to give preferred answers a higher score.
  3. The assistant generates answers and is trained toward those with higher reward scores. This is reinforcement learning: changing behaviour using a score.

InstructGPT used this recipe, with an algorithm called proximal policy optimization (PPO) for the last stage (Ouyang et al. 2022). The reward model's score stands in for people's judgments; a high score alone does not prove an answer is useful.

Training can push the assistant too far from useful behaviour it already had. A reference model is an unchanged copy kept as a comparison point. A stay-close penalty discourages large changes in answer probabilities relative to that copy. This is called a Kullback-Leibler penalty, or KL penalty. It limits drift, but is not a guarantee of safety or quality (Ouyang et al. 2022).

learns directly from a preferred and a rejected answer to the same prompt. Its original form uses a frozen reference model but skips both a separately trained reward model and the reinforcement-learning loop that generates fresh answers during training (Rafailov et al. 2023). Its loss is classification-like, and a setting called beta controls the strength of the reference constraint. In one controlled comparison, preference-data quality mattered more than the choice of algorithm (Ivison et al. 2024).

Contested: DPO or PPO? The DPO paper reports matching or beating PPO on its tasks. Later controlled studies found well-tuned PPO ahead: in all cases they tested, in one (Xu et al. 2024), and by up to 2.5% on maths and 1.2% in general, in another (Ivison et al. 2024). DPO stays popular because it is cheap and stable.

When a program can check an answer, such as code against tests or a maths answer against a key, its score can supply the reward. This is reinforcement learning with verifiable rewards. Group relative policy optimization () generates several answers to one prompt and compares their scores with their group's average. Above-average answers get a positive learning signal; below-average answers get a negative one. The comparison follows the chosen scorer, which may be imperfect; reward is not independent proof of truth (Shao et al. 2024).

Illustration: four answers score 1, 0, 0 and 1. Their average is 0.5. Using the population standard deviation as the spread gives 0.5. Subtract the average from each score and divide by that spread to get the relative signals. Implementations can use different scaling conventions; the example shows the direction and relative comparison.

Answer1234
Scorer's reward1001
Relative signal+1−1−1+1

If all four scores are equal, none is above or below the average. The relative reward signal is then zero for every answer; implementations handle the zero spread rather than dividing by zero. There is no reward preference to learn from that group. Other terms, such as an enabled KL penalty, can still contribute to updates (TRL docs). DeepSeek-R1-Zero used GRPO without a supervised fine-tuning stage first and rose from 15.6% to 77.9% on the AIME 2024 maths problems during training (DeepSeek-AI 2025).

Every one of these methods optimises a stand-in for what people actually want, and stand-ins can be gamed. is when the model scores highly without doing the task. DeepSeek saw it with a learned helpfulness reward and limited how long it trained on that signal (DeepSeek-AI 2025). Pushed hard against a learned reward model, true quality first rises and then falls (Gao et al. 2022). Sycophancy, telling users what they want to hear, has been linked to preference training (Sharma et al. 2023), and in April 2025 OpenAI rolled back a GPT-4o update that had become sycophantic (OpenAI). Preference training also tends to make answers longer, because raters, reward models and judges often favour length (Singhal et al. 2023).

The usual order is supervised fine-tuning first, then preference tuning or reinforcement learning, because these methods refine answers the model can already produce. OpenAI's and Google's preference tuning guides both advise it (OpenAI, Google Cloud).

Going deeper

PPO commonly uses a separate value model to estimate expected reward. GRPO replaces that model with comparisons within each sampled group, reducing memory use in the original method (Shao et al. 2024). For DPO, TRL 1.16.0 used a beta default of 0.1 when checked on 28 October 2026; consult the version you run rather than treating that as a universal setting (TRL docs).

DPO's relatives change the data or drop the reference model: KTO learns from single answers marked good or bad (Ethayarajh et al. 2024), and ORPO merges supervised and preference training into one stage (Hong et al. 2024).

Contested: does reinforcement learning add new ability? DeepSeekMath found RL improved maj@k (the most common of k answers is right) but not pass@k (any of k answers is right), suggesting it made existing solutions more reliable rather than adding new ones (Shao et al. 2024); another study found base models win at large k (Yue et al. 2025). Prolonged RL reports the opposite (Liu et al. 2025). Unresolved. The "aha moment" of self-reflection reported for R1-Zero also appears in the base model before RL, according to a follow-up (Liu et al. 2025).

Whether the KL penalty is needed is debated: in one controlled study it did not measurably improve true reward (Gao et al. 2022), DAPO drops it for long reasoning (Yu et al. 2025), and TRL 1.16.0's GRPO default sets it to zero, as of 28 October 2026. Online methods, which judge fresh answers from the current model, beat offline ones in reward over-optimisation experiments (Tang et al. 2024). AI feedback can replace human labels with comparable results on tested tasks (Lee et al. 2023), as in Constitutional AI (Bai et al. 2022). Even a checker can be gamed, since tests can be incomplete: reasoning models were caught reward hacking in coding tasks, and penalising "bad thoughts" under strong optimisation taught them to hide their intent while still hacking (Baker et al. 2025).

Managed reinforcement fine-tuning, as of 28 October 2026: OpenAI's guide lists only o4-mini, which it says shut down on 23 October 2026, inside a platform that is winding down (OpenAI); Amazon Bedrock trains with GRPO on Nova 2 Lite, gpt-oss-20B and Qwen3 32B and advises starting with 100 to 200 examples (AWS); Google Cloud's RL tuning for Gemini is marked pre-GA, not for production (Google Cloud).

Shipping it and keeping it working

If you trained a LoRA adapter, you have two ways to serve it. Merging adds the correction into the base weights, leaving one model with no separate adapter calculation (Hu et al. 2021). Keeping the separate lets a compatible server share one base model across variants. Choose for your workload and test the speed, rather than treating either arrangement as always faster. An adapter needs the base model it was trained for because it stores a correction to those weights. Its configuration records the base model name and, when set, revision (PEFT). A changed base model requires retraining or validating a new adapter; do not assume the old correction transfers.

Using the trained model to produce an answer is inference. Serving is the surrounding work of accepting requests, sharing hardware and returning those answers. Start with what happens during inference, then read how models are served for batching, caches and serving engines.

Quantization makes a model smaller to serve: weights normally stored in 16 bits are rounded to 8 or 4, so the model needs less memory and can run on smaller hardware, even a laptop. How much quality that costs is contested: GPTQ reports negligible loss at 3 to 4 bits on very large models (Frantar et al. 2022), while a study of LLaMA3 found non-negligible degradation, especially at very low bit widths (Huang et al. 2024). Quantize the exact model you will serve, then re-run the same tests, safety included.

If you trained a LoRA adapter and will both merge it and quantize, merge the correction into the base weights first, then quantize the merged model. Quantizing can shift quality, so those tests have to be run again on the quantized file, not only on the model you trained. Keeping the adapter separate is a different setup: test that arrangement instead of assuming the merged numbers still apply.

An open-weight model comes with a licence, and a model you fine-tune from it is usually a derivative that inherits its conditions. The Llama 3.1 licence, for example, asks anyone distributing a model trained with Llama materials or outputs to put "Llama" at the beginning of its name (Llama 3.1 licence). Google moved Gemma 4 to the Apache 2.0 licence in April 2026, while earlier Gemma models stay under the Gemma Terms of Use (Google, Gemma terms). Read the licence for the exact version you use. This is not legal advice.

Before relying on a provider, check whether it supports your exact model and tuning method, how long the tuned model can be served, and whether the result can be exported. Keep the dataset and evaluation tests so you can move later. Provider support changes; use its current documentation when making the decision. The reference note records the availability last checked for this piece.

Provider availability: dated reference note

Checked 28 October 2026. This snapshot is background; follow each provider's current documentation before starting a job.

  • OpenAI says its fine-tuning platform is winding down: closed to new users, with existing eligible customers able to create jobs until 6 January 2027. Tuned models follow their base model's shutdown schedule (OpenAI, deprecations).
  • Google: the Gemini API and AI Studio offer no tuning (Gemini API); tuning Gemini models happens on Google Cloud, which documents supervised, preference and reinforcement tuning (Google Cloud).
  • Amazon Bedrock offers supervised fine-tuning, reinforcement fine-tuning and distillation (AWS).
  • Mistral's fine-tuning documentation is marked deprecated and no longer actively supported (Mistral).
  • Together AI and Fireworks document LoRA and full fine-tuning of open-weight models, with supervised and DPO training; Together says tuned weights can be downloaded (Together AI, Fireworks).
  • Self-hosted: with an open-weight model you can train on your own or rented GPUs with open libraries such as TRL and PEFT, and keep the result (TRL).

Testing does not stop at launch.

Monitor
Errors, speed, cost, format failures and a sample of quality scores on real traffic, and grow your test set with real failures; OpenAI's evaluation guide recommends logging everything and running evals on every change (OpenAI). Inputs drift as products and users change, and a model that tested well can quietly get worse (Rabanser et al. 2018).
Version
Everything: base model and revision, adapter, training data, settings and seed, test set and scorer, quantization and licence.
Retrain
On evidence: when quality falls, inputs change, the task changes, or a better base model arrives. When one does, run the baselines again, because the new model with a good prompt may already beat your old fine-tune.
Going deeper

The vLLM serving engine documents per-request adapters with minimal overhead (vLLM), and the S-LoRA paper reports serving thousands of adapters on a single GPU (Sheng et al. 2023).

AWQ, another common quantization method, protects the small share of weights that matter most (Lin et al. 2023), and GGUF is a single-file format used by llama.cpp and similar tools that run models on ordinary hardware (Hugging Face). If you trained an adapter, decide whether to merge it before converting, and test the file you ship at the bit width you ship. Latency is how long one user waits, including time to first token; throughput is total output across all users, and raising concurrency trades one for the other (NVIDIA). Fine-tuning that changes answer length changes latency and cost too. Hosted training is usually billed on training tokens, dataset size times epochs; Amazon Bedrock also charges monthly storage per custom model (AWS). Prices change often, so take them from the provider's page on the day.

Keep the dataset and the evaluation harness, not just the tuned model: they are what make moving to a new base model cheap. Model cards document intended use and evaluation results (Mitchell et al. 2018), and datasheets document how a dataset was built (Gebru et al. 2018). With open weights you can keep running an old model as long as you like, at the cost of serving it yourself.

Review the key ideas

Mini glossary

Fine-tuning
training an already trained model further on your own examples, so its weights change.
Supervised fine-tuning (SFT)
training on pairs of inputs and the exact outputs you want.
Chat template
the model-specific recipe that turns a list of messages into the token string the model reads.
Loss masking
choosing which tokens contribute to loss; assistant-only training scores the reply while still reading the prompt.
Held-out test set
examples kept aside and never trained on or tuned against, used for the final check.
Learning rate
how big a step each weight update takes.
Epoch
one full pass over the training data; a step is one weight update. More passes raise overfitting and forgetting risk.
Overfitting
fitting training examples too closely so performance on new inputs suffers.
Catastrophic forgetting
losing earlier skills while training on something new.
LoRA
low-rank adaptation: train a small correction beside unchanged base weights, saving training memory.
QLoRA
train a LoRA adapter while storing the unchanged base weights in 4-bit form; calculations use higher precision.
DPO
direct preference optimization: training on chosen and rejected pairs without a reward model.
GRPO
reinforcement learning that scores each answer against a group of answers to the same prompt.
Reward hacking
scoring highly by exploiting flaws in the reward instead of doing the task.

Sources and further reading

The original sources were fetched on 28 October 2026, with three additions marked 29 October. Loss, retrieval and GRPO explanations were rechecked during the readability update. Vendor features, defaults and prices change; follow the links for current details.

  1. Brown et al., "Language Models are Few-Shot Learners" (2020). arxiv.org/abs/2005.14165
  2. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020). arxiv.org/abs/2005.11401
  3. Google Cloud, "Introduction to tuning" (Gemini). docs.cloud.google.com/vertex-ai/generative-ai/docs/models/tune-models
  4. Mistral, "Fine-tuning" (marked deprecated). docs.mistral.ai/capabilities/finetuning
  5. OpenAI, "Model optimization". developers.openai.com/api/docs/guides/model-optimization
  6. Ovadia et al., "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs" (2023). arxiv.org/abs/2312.05934
  7. Gekhman et al., "Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?" (EMNLP 2024). arxiv.org/abs/2405.05904
  8. Asai et al., "Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection" (2023). arxiv.org/abs/2310.11511
  9. Gururangan et al., "Don't Stop Pretraining" (2020). arxiv.org/abs/2004.10964
  10. Ouyang et al., "Training language models to follow instructions with human feedback" (2022), InstructGPT. arxiv.org/abs/2203.02155
  11. Allen-Zhu and Li, "Physics of Language Models: Part 3.1" (2023). arxiv.org/abs/2309.14316
  12. Mecklenburg et al., "Injecting New Knowledge into Large Language Models via Supervised Fine-Tuning" (2024). arxiv.org/abs/2404.00213
  13. OpenAI, "Supervised fine-tuning". developers.openai.com/api/docs/guides/supervised-fine-tuning
  14. Hugging Face Transformers, "Chat templates". huggingface.co/docs/transformers/main/en/chat_templating
  15. Hugging Face TRL, "SFT Trainer" (version 1.16). huggingface.co/docs/trl/sft_trainer
  16. Shi et al., "Instruction Tuning With Loss Over Instructions" (2024). arxiv.org/abs/2405.14394
  17. Zhou et al., "LIMA: Less Is More for Alignment" (2023). arxiv.org/abs/2305.11206
  18. Dettmers et al., "QLoRA: Efficient Finetuning of Quantized LLMs" (2023). arxiv.org/abs/2305.14314
  19. Google Cloud, "Prepare supervised fine-tuning data for Gemini models". docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini-supervised-tuning-prepare
  20. Gudibande et al., "The False Promise of Imitating Proprietary LLMs" (2023). arxiv.org/abs/2305.15717
  21. OpenAI, Terms of Use (effective 1 January 2026). openai.com/policies/row-terms-of-use
  22. Lee et al., "Deduplicating Training Data Makes Language Models Better" (2021). arxiv.org/abs/2107.06499
  23. scikit-learn, "Cross-validation: evaluating estimator performance". scikit-learn.org/stable/modules/cross_validation.html
  24. Carlini et al., "Extracting Training Data from Large Language Models" (2020). arxiv.org/abs/2012.07805
  25. European Data Protection Board, statement on Opinion 28/2024 on AI models (18 December 2024). edpb.europa.eu
  26. Raghavendra et al., "Revisiting the Superficial Alignment Hypothesis" (2024). arxiv.org/abs/2410.03717
  27. Stanford CRFM, "Alpaca: A Strong, Replicable Instruction-Following Model" (2023). crfm.stanford.edu/2023/03/13/alpaca.html
  28. Shumailov et al., "The Curse of Recursion: Training on Generated Data Makes Models Forget" (2023). arxiv.org/abs/2305.17493
  29. Gerstgrasser et al., "Is Model Collapse Inevitable?" (2024). arxiv.org/abs/2404.01413
  30. Sainz et al., "NLP Evaluation in trouble" (2023), on benchmark contamination. arxiv.org/abs/2310.18018
  31. Hugging Face Transformers, "Trainer" and TrainingArguments. huggingface.co/docs/transformers/main/en/main_classes/trainer
  32. Unsloth, "LoRA hyperparameters guide". unsloth.ai/docs/get-started/fine-tuning-llms-guide/lora-hyperparameters-guide
  33. Biderman et al., "LoRA Learns Less and Forgets Less" (TMLR 2024). arxiv.org/abs/2405.09673
  34. Google Machine Learning Crash Course, "Overfitting". developers.google.com/machine-learning/crash-course/overfitting/overfitting
  35. Microsoft Foundry, "Customize a model with fine-tuning". learn.microsoft.com/en-us/azure/foundry/openai/how-to/fine-tuning
  36. Goodfellow, Bengio and Courville, Deep Learning (2016), chapter 10, section on teacher forcing. deeplearningbook.org/contents/rnn.html
  37. Bengio et al., "Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks" (2015). arxiv.org/abs/1506.03099
  38. Ranzato et al., "Sequence Level Training with Recurrent Neural Networks" (ICLR 2016), which names exposure bias. arxiv.org/abs/1511.06732
  39. Luo et al., "An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning" (2023). arxiv.org/abs/2308.08747
  40. Scialom et al., "Fine-tuned Language Models are Continual Learners" (2022), Continual-T0. arxiv.org/abs/2205.12393
  41. Hugging Face Transformers, "Text generation", fetched 29 October 2026. huggingface.co/docs/transformers/main/en/llm_tutorial
  42. Holtzman et al., "The Curious Case of Neural Text Degeneration" (ICLR 2020), fetched 29 October 2026. arxiv.org/abs/1904.09751
  43. Hugging Face PEFT, "PEFT checkpoint format", fetched 29 October 2026. huggingface.co/docs/peft/main/en/developer_guides/checkpoint
  44. Hugging Face Transformers, "Perplexity of fixed-length models". huggingface.co/docs/transformers/main/en/perplexity
  45. PyTorch 2.14, "Reproducibility". docs.pytorch.org/docs/2.14/notes/randomness.html
  46. Hugging Face Transformers, "GPU memory usage" (model memory anatomy). huggingface.co/docs/transformers/main/en/model_memory_anatomy
  47. Ben-Zaken, Ravfogel and Goldberg, "BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models" (2021). arxiv.org/abs/2106.10199
  48. Hugging Face PEFT documentation. huggingface.co/docs/peft/index
  49. Hugging Face PEFT, "LoRA" configuration reference. huggingface.co/docs/peft/main/en/package_reference/lora
  50. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021). arxiv.org/abs/2106.09685
  51. Kalajdzievski, "A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA" (2023), rsLoRA. arxiv.org/abs/2312.03732
  52. Liu et al., "DoRA: Weight-Decomposed Low-Rank Adaptation" (ICML 2024). arxiv.org/abs/2402.09353
  53. Houlsby et al., "Parameter-Efficient Transfer Learning for NLP" (2019), adapters. arxiv.org/abs/1902.00751
  54. Micikevicius et al., "Mixed Precision Training" (2017). arxiv.org/abs/1710.03740
  55. Chen et al., "Training Deep Nets with Sublinear Memory Cost" (2016), gradient checkpointing. arxiv.org/abs/1604.06174
  56. Dettmers et al., "8-bit Optimizers via Block-wise Quantization" (2021). arxiv.org/abs/2110.02861
  57. DeepSpeed, "Zero Redundancy Optimizer" tutorial. deepspeed.ai/tutorials/zero
  58. torchtune on GitHub (README: no longer actively maintained). github.com/pytorch/torchtune
  59. Anthropic, "Define your success criteria". platform.claude.com/docs/en/test-and-evaluate/define-success
  60. Miller, "Adding Error Bars to Evals" (2024). arxiv.org/abs/2411.00640
  61. Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023). arxiv.org/abs/2306.05685
  62. Wang et al., "Large Language Models are not Fair Evaluators" (2023). arxiv.org/abs/2305.17926
  63. Dubois et al., "Length-Controlled AlpacaEval" (2024). arxiv.org/abs/2404.04475
  64. Panickssery et al., "LLM Evaluators Recognize and Favor Their Own Generations" (2024). arxiv.org/abs/2404.13076
  65. Brown, Cai and DasGupta, "Interval Estimation for a Binomial Proportion" (Statistical Science, 2001). doi.org/10.1214/ss/1009213286
  66. Dodge et al., "Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping" (2020). arxiv.org/abs/2002.06305
  67. Qi et al., "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" (2023). arxiv.org/abs/2310.03693
  68. Röttger et al., "XSTest" (2023), exaggerated safety. arxiv.org/abs/2308.01263
  69. Yang et al., "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples" (2023). arxiv.org/abs/2311.04850
  70. Gao et al., "Scaling Laws for Reward Model Overoptimization" (2022). arxiv.org/abs/2210.10760
  71. Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (2023). arxiv.org/abs/2305.18290
  72. Hugging Face TRL, "DPO Trainer". huggingface.co/docs/trl/dpo_trainer
  73. Ethayarajh et al., "KTO: Model Alignment as Prospect Theoretic Optimization" (2024). arxiv.org/abs/2402.01306
  74. Hong et al., "ORPO: Monolithic Preference Optimization without Reference Model" (2024). arxiv.org/abs/2403.07691
  75. Ivison et al., "Unpacking DPO and PPO" (2024). arxiv.org/abs/2406.09279
  76. Xu et al., "Is DPO Superior to PPO for LLM Alignment?" (2024). arxiv.org/abs/2404.10719
  77. Shao et al., "DeepSeekMath" (2024), which introduced GRPO. arxiv.org/abs/2402.03300
  78. DeepSeek-AI, "DeepSeek-R1" (2025). arxiv.org/abs/2501.12948
  79. Sharma et al., "Towards Understanding Sycophancy in Language Models" (2023). arxiv.org/abs/2310.13548
  80. OpenAI, "Expanding on what we missed with sycophancy" (2025). openai.com/index/expanding-on-sycophancy
  81. Singhal et al., "A Long Way to Go: Investigating Length Correlations in RLHF" (2023). arxiv.org/abs/2310.03716
  82. OpenAI, "Direct preference optimization". developers.openai.com/api/docs/guides/direct-preference-optimization
  83. Google Cloud, "Preference tuning" for Gemini. docs.cloud.google.com/vertex-ai/generative-ai/docs/models/gemini-preference-tuning
  84. Yue et al., "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" (2025). arxiv.org/abs/2504.13837
  85. Liu et al., "ProRL: Prolonged Reinforcement Learning" (2025). arxiv.org/abs/2505.24864
  86. Liu et al., "Understanding R1-Zero-Like Training: A Critical Perspective" (2025), Dr. GRPO. arxiv.org/abs/2503.20783
  87. Yu et al., "DAPO" (2025). arxiv.org/abs/2503.14476
  88. Tang et al., "Understanding the performance gap between online and offline alignment algorithms" (2024). arxiv.org/abs/2405.08448
  89. Lee et al., "RLAIF vs. RLHF" (2023). arxiv.org/abs/2309.00267
  90. Bai et al., "Constitutional AI: Harmlessness from AI Feedback" (2022). arxiv.org/abs/2212.08073
  91. Baker et al., "Monitoring Reasoning Models for Misbehavior" (2025). arxiv.org/abs/2503.11926
  92. OpenAI, "Reinforcement fine-tuning". developers.openai.com/api/docs/guides/reinforcement-fine-tuning
  93. Amazon Bedrock, "Reinforcement fine-tuning". docs.aws.amazon.com/bedrock/latest/userguide/reinforcement-fine-tuning.html
  94. Google Cloud, "Reinforcement tuning" (pre-GA). docs.cloud.google.com/gemini-enterprise-agent-platform/models/tuning/reinforcement-tuning
  95. vLLM, "LoRA adapters". docs.vllm.ai/en/latest/features/lora.html
  96. Sheng et al., "S-LoRA: Serving Thousands of Concurrent LoRA Adapters" (2023). arxiv.org/abs/2311.03285
  97. Frantar et al., "GPTQ" (2022). arxiv.org/abs/2210.17323
  98. Lin et al., "AWQ: Activation-aware Weight Quantization" (2023). arxiv.org/abs/2306.00978
  99. Hugging Face Hub, "GGUF". huggingface.co/docs/hub/gguf
  100. Huang et al., "How Good Are Low-bit Quantized LLaMA3 Models?" (2024). arxiv.org/abs/2404.14047
  101. Meta, Llama 3.1 Community License Agreement. github.com/meta-llama/llama-models
  102. Google Open Source Blog, "Gemma 4: expanding the Gemmaverse with Apache 2.0" (2026). opensource.googleblog.com
  103. Google, Gemma Terms of Use. ai.google.dev/gemma/terms
  104. OpenAI, "Deprecations". developers.openai.com/api/docs/deprecations
  105. Google AI for Developers, "Gemini API model tuning". ai.google.dev/gemini-api/docs/model-tuning
  106. Amazon Bedrock, "Customize your model". docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html
  107. Together AI, "Fine-tuning overview". docs.together.ai/docs/fine-tuning-overview
  108. Fireworks, "Fine-tuning models". docs.fireworks.ai/fine-tuning/fine-tuning-models
  109. Hugging Face TRL documentation. huggingface.co/docs/trl/index
  110. OpenAI, "Evaluation best practices". developers.openai.com/api/docs/guides/evaluation-best-practices
  111. Rabanser et al., "Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift" (2018). arxiv.org/abs/1810.11953
  112. NVIDIA NIM, "Benchmarking metrics". docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
  113. Mitchell et al., "Model Cards for Model Reporting" (2018). arxiv.org/abs/1810.03993
  114. Gebru et al., "Datasheets for Datasets" (2018). arxiv.org/abs/1803.09010

Want every term in one place?

Fine-tuning, every concept

Hear when the next piece is ready

New long reads and explainers on aidojo.si. We only write when something new is ready.

Your address goes to Buttondown, the service that sends AI Dojo emails. You confirm it by email first, and every email has a link to unsubscribe.