Transformer basics

What loss does language-model pretraining minimize?

Cross-entropy: the negative log probability the model gives the correct next token, averaged over positions.

What is perplexity?

The exponential of the average per-token cross-entropy. Lower is better. It only compares models that share a tokenizer and test set.

Why report bits per byte instead of perplexity?

It divides total loss by the number of bytes of text, not tokens, so models with different tokenizers can be compared on the same text.

What is teacher forcing?

During training, each position is conditioned on the true previous tokens, not on the model's own earlier predictions.

What is exposure bias?

The gap between training, where the model always sees true prefixes, and generation, where it sees its own possibly wrong outputs.

What is masked language modeling?

Hide a fraction of input tokens and train the model to recover them using context on both sides. Used by encoder models such as BERT.

What is span corruption?

Replace contiguous spans of input with sentinel tokens and train the model to output the missing spans. Used by T5.

What is a prefix language model?

A decoder where the prefix is attended bidirectionally and only the continuation uses a causal mask and is scored.

In self-attention, where do queries, keys, and values come from?

From the same sequence: each is a separate learned linear projection of the layer's input vectors.

After softmax, what is one attention row for a single query?

A probability distribution over the allowed key positions: non-negative weights that sum to one.

Why are attention scores divided by √d_k?

Dot products of random d_k-dimensional vectors grow with √d_k. Without scaling, softmax saturates and gradients become tiny.

What can a byte-level vocabulary represent without an unknown-token bucket?

Any input string, because every string is a sequence of bytes and all 256 bytes are in the base vocabulary.

What is tokenizer fertility?

The average number of tokens per word. Higher fertility on a language means longer sequences and fewer words per context window.

How does RoPE encode position?

It rotates query and key vectors by angles proportional to position, so their dot product depends on the relative offset.

What is length extrapolation?

Running a model on sequences longer than any it was trained on. Many position schemes degrade there without changes such as RoPE scaling.

When is dropout active?

Only in training. At inference it is turned off, and with inverted dropout no rescaling is needed then.

What does weight decay do?

Shrinks weights toward zero at each step, a form of regularization. Norm and bias parameters are often excluded.

How does AdamW differ from Adam with L2 regularization?

AdamW applies weight decay directly to the weights, separate from the adaptive gradient update, instead of adding it to the gradient.

What is learning-rate warmup?

Raising the learning rate from near zero over the first steps, which avoids instability while optimizer statistics are still poor.

What is a cosine learning-rate schedule?

After warmup, the learning rate decays along a half cosine curve from its peak to a small final value.

What does gradient clipping do?

Rescales the gradient when its global norm passes a threshold, which limits the size of any single update.

What is a loss spike?

A sudden jump in training loss, often from a bad batch or unstable optimizer state. Common fixes are skipping batches or restarting from a checkpoint.

Why is bfloat16 popular for training?

It keeps float32's exponent range with fewer mantissa bits, so it rarely overflows and usually needs no loss scaling.

What is mixed-precision training?

Most math runs in 16-bit, while master weights and some reductions stay in 32-bit to keep updates accurate.

Why are residual output projections often initialized smaller?

Each layer adds to the residual stream. Scaling init by depth keeps the stream's variance from growing with the number of layers.

Why do most large models use pre-norm?

Normalizing inside each branch leaves a clean residual path, which trains deep stacks more stably and needs less warmup.

What is sequence packing?

Concatenating several short documents into one training sequence to avoid wasted padding.

What must change in the mask when sequences are packed?

If documents should not see each other, the mask must block attention across document boundaries, not just future positions.

What are tied embeddings?

The input embedding matrix is reused, transposed, as the output projection, which saves parameters.

About how many FLOPs does training a dense transformer cost?

Roughly 6 × parameters × training tokens: about 2 for the forward pass and 4 for the backward pass.

What do neural scaling laws describe?

Smooth power-law relations between loss and model size, data, or compute, which let small runs predict larger ones.

What was the Chinchilla finding?

For a fixed compute budget, parameters and training tokens should grow together, about 20 tokens per parameter. Many earlier models were undertrained.

Why is large-model pretraining often about one epoch?

With enough data, fresh tokens help more than repeats. Repeating data many times raises the risk of memorization.

Why deduplicate pretraining data?

Duplicates waste compute, increase memorization, and can leak test examples into training.

What is benchmark contamination?

Test items, or close copies of them, appearing in training data, which inflates the reported score.

What is supervised fine-tuning (SFT)?

Further training a pretrained model on prompt and response pairs with the usual next-token loss, often scored on response tokens only.

What is a reward model in RLHF?

A model trained on human preference pairs to output a scalar score, which then guides policy optimization.

What does a KL term do in a preference update?

It penalizes the policy for drifting from a reference model, which limits reward hacking and keeps outputs fluent.

What is the reference model in RLHF or DPO?

A frozen copy, usually the SFT model, that the trained policy is compared against through the KL term or log ratio.

What is direct preference optimization (DPO)?

A loss that trains on preference pairs directly, raising the log ratio of chosen over rejected responses versus the reference, without a reward model.

What is knowledge distillation?

Training a smaller student model to match a larger teacher's output distribution, not just the hard labels.

Why does a mixture-of-experts model need a load-balancing loss?

Without it the router can send most tokens to a few experts, leaving the others undertrained and overloading some devices.

What is an auxiliary z-loss?

A small penalty on the log of the softmax normalizer, which keeps logits from growing large and improves stability.

What is in-context learning?

A model doing a new task from examples or instructions in the prompt, with no change to its weights.

What is an induction head?

An attention pattern that finds an earlier occurrence of the current token and copies the token that followed it.

What is a linear probe?

A simple classifier trained on frozen hidden states to test whether some property is linearly readable from them.

What is the logit lens?

Applying the final unembedding to intermediate residual-stream states to see what each layer would predict.

What is an ablation?

Removing or disabling one component, holding the rest fixed, and measuring the change in results.

Why track held-out validation loss during pretraining?

Training loss on fresh data already estimates generalization, but a fixed held-out set gives a stable curve to compare across runs.