Card 1 of 50
Front. Baseline. What is the baseline before a fine-tune?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 50 cards in this deck
1.Baseline
What is the baseline before a fine-tune?
The current prompt on the frozen model, scored on the same holdout you will use for the fine-tune. The fine-tune has to clear that number to justify the extra system.
2.One-change
What do you change between two runs?
One factor: the data, the objective, or the adapter. A run that changes all three cannot say which one moved the score.
3.Prompt-first
When do you stop at prompting?
When the prompt baseline already clears the bar on the holdout, including the format. Fine-tuning is for the failures that stay after that.
4.Sft-when
When is SFT the experiment?
When you have trusted input and output pairs and you need the model to imitate that format or style. The loss is next-token on the target completion.
5.Dpo-when
When is DPO the next experiment after SFT?
When you have pairs of better and worse answers to the same prompt, and a reference checkpoint. It is a comparison loss, not more imitation.
6.Grpo-when
When is a group-relative method such as GRPO the experiment?
When you can score several samples of one prompt, as in a verifiable reasoning task, and you do not have a trained reward model.
7.Reward-when
When do you train a reward model at all?
When the preference signal is expensive to recompute and you will reuse the score. If the judge is a unit test, score the samples directly.
8.Rows-sft
What does one SFT row contain?
The conversation up to the assistant's target, rendered with the chat template you will serve. The loss mask covers the assistant tokens only.
9.Rows-dpo
What does one DPO row contain?
One prompt, one chosen completion, and one rejected completion. Both completions go through the same template. The reference is the model you are moving away from.
10.Ref-pin
What do you pin as the reference?
The exact revision of the SFT or base checkpoint the policy starts from. A newer reference makes the KL term a different experiment.
11.Template-exp
Why is the chat template part of the study?
It decides which tokens exist. Training with one template and scoring with another measures the template mismatch, not the data.
12.Mask
What happens if the loss includes the prompt tokens?
The model spends capacity relearning the user's text. Mask the prompt. Report that the mask was on, because the loss value is not comparable without it.
13.Pack-exp
What do you check if you pack short rows?
That attention and the loss cannot cross from one example into the next. A higher token throughput with a leaky mask is a contaminated gradient.
14.Split
How do you split adaptation data?
By the unit that would leak, such as the user or the document, not by a random row. The holdout is untouched until the run you will report.
15.Contamination
What contamination do you look for?
Holdout prompts, or near copies, inside the fine-tune rows, and eval items that were in the base model's pretraining. Either one inflates the score.
16.Datasets-fp
What do you record from the datasets library?
The dataset revision and the fingerprint of the map that built the rows. A cached map can hide a preprocessing change you thought you had rerun.
17.Synth
How do you use synthetic completions?
As a draft the teacher model wrote, then filter them with a rule or a person on a sample. Log the teacher revision. The student will copy the teacher's mistakes.
18.Teacher
What is the teacher in a distillation study?
The model that wrote the targets. Its revision is a factor in the experiment, the same way the learning rate is.
19.Judge
When is an LLM judge the metric?
When the answer is open-ended and you have checked the judge against human labels on this task. A judge that prefers longer answers will crown the longer policy.
20.Judge-blind
How do you keep the judge from seeing the condition?
Shuffle order, hide which run produced the answer, and score both with the same rubric. A judge that knows the name of the method is grading the name.
21.Hard-set
What belongs in the holdout besides happy prompts?
Cases the model should refuse, cases with a false premise, and cases just outside the format. A fine-tune can win the happy set and lose those.
22.Slice
Why report slices?
An average can rise because one easy slice moved. The slice you care about, such as the long prompts or the rare class, gets its own number.
23.Stop
When do you stop training?
When the holdout metric you set in advance peaks, or when a preset token budget is spent. Training loss can keep falling after that holdout has already peaked.
24.Epochs
Why are a few epochs enough on a small SFT set?
The model sees each target many times. Past the peak, it memorizes phrasing. The holdout curve shows the turn.
25.Lr
What do you sweep before you sweep architecture?
Learning rate and the number of tokens. Those move the result more cheaply than a new base model. Record the sweep, including the values that were worse.
26.Rank
What does a LoRA rank sweep tell you?
Whether the adapter has enough capacity. If rank 8 and rank 64 tie on the holdout, the limit is the data, not the adapter width.
27.Targets
Which modules do you adapt first?
The attention projections the recipe defaults to. Add the MLP only if the holdout is still short of the bar. Adapting every module spends the memory first.
28.Qlora-exp
When is QLoRA the right study setup?
When the full base does not fit and you accept a possible quality gap versus a 16-bit LoRA. Score both if you can fit the 16-bit run even once.
29.Unsloth-exp
When do you use an accelerated LoRA kernel in a study?
To iterate. The run you report should also load in the stack you will serve, so a kernel-only checkpoint is a dead end.
30.Seed
Why more than one seed?
A single seed can win by the order of a small dataset. Two or three seeds, with the spread reported, show whether the gain is larger than the noise.
31.Tokens-seen
What do you align when you compare runs?
Tokens seen, or optimizer steps at the same batch in tokens. A run that took more steps has not necessarily learned more if the sequences were shorter.
32.Eval-gen
Why score generations and not only loss?
The product samples text. A lower loss can still format badly or refuse more. Use the same decoder settings you will serve.
33.Decode
What decoding settings do you freeze?
Temperature, top-p, and max tokens, written in the report. A gain that appears only at a new temperature is a decoding result.
34.Regression
What is the regression set for an adaptation?
Prompts the previous model handled, which the new one must still handle. A domain fine-tune that breaks the general set needs that number beside the domain win.
35.Forget
How do you notice catastrophic forgetting?
The regression set drops while the in-domain set rises. That pair of curves is the result. The in-domain number alone hides the cost.
36.Merge
When do you merge the adapter before the final score?
When serving will merge it. Score the merged weights. An adapter-on-base score can differ once the weights are folded in.
37.Base-choice
How do you choose the base checkpoint?
The instruction-tuned model if you need to keep chat behavior, the base model if you will teach the format yourself. Name the revision in the report.
38.Track
What does the experiment tracker have to contain?
The config, the data revision, the seed, the eval scores, and the link to the checkpoint. A loss plot without those cannot be cited.
39.Mlflow-vs-wb
Which tracker do you use for the study?
Either a run tracker such as Weights & Biases or a registry-backed log such as MLflow, as long as the config and the holdout score are on the same run id.
40.Negative
What do you do with a run that loses to the prompt?
Report it, with the hypothesis it killed. A study that only shows the winning fine-tune hides how narrow the gain was.
41.Cost-exp
What cost do you report beside the score?
GPU hours and the inference cost of the resulting model. A small quality gain that doubles serving cost is a different decision.
42.Data-card
What is on the data card for the fine-tune set?
Who wrote the rows or which teacher did, the filter rules, the split unit, and the known gaps. The score is not interpretable without that.
43.License-data
What do you check on the rows before training?
That the license allows training and allows shipping a model that imitated those rows. Record the license next to the dataset revision.
44.Pii
What do you strip from adaptation rows?
Secrets, account identifiers, and text the product must not memorize. A fine-tune repeats its rows more readily than pretraining does.
45.Leak-prod
How can a fine-tune leak into the product?
By memorizing a row and emitting it. Deduplicate, and test a membership-style probe on a sample of training strings.
46.Online
When do you collect the next dataset from production?
After the current model is live, with a log of what the user saw, and with consent and retention rules. Those logs are a new dataset, not a free extension of the old one.
47.Offline-first
Why finish the offline study before an online test?
The offline holdout is cheap and catches a format break. The online test spends users and measures what the holdout cannot, such as retention.
48.Ab-adapt
What is the unit of an online test of a fine-tune?
The user, or the session, assigned once. Scoring by request lets one user land in both models and shrinks the variance for the wrong reason.
49.Report
What does the written result contain?
The prompt baseline, the chosen run, the holdout and slices, the seeds, the token budget, and the checkpoint id, enough to reload that checkpoint and score it again.
50.Stop-study
When is the study done?
When the decision you named at the start is answered: ship the checkpoint, keep the prompt, or collect a specific kind of row. More runs after that need a new question.