Transformer basics

Question 1

The attention score matrix for one head grows with which power of the sequence length, and which other factors multiply that memory?

Accepted Answer

It is sequence by sequence, so it grows with the square of the length, and also with the batch and the head count.

Question 2

What is the shape of the token embedding table, and what does a longer sequence or a larger vocabulary do to the score matrix?

Accepted Answer

The table is vocabulary by d_model. A longer sequence does not widen it, and a larger vocabulary does not grow the score matrix.

Question 3

How does feed-forward activation memory scale with sequence length?

Accepted Answer

It grows linearly with the sequence length.

Question 4

Besides sequence length, which width sets the feed-forward activation's size?

Accepted Answer

The intermediate width. That activation is batch by sequence by the intermediate width.

Question 5

Which attention tensor is quadratic in the sequence length?

Accepted Answer

The per-head score matrix. The value vectors stay linear in the length.

Question 6

A causal mask blocks future positions. What does that do to the number of score pairs?

Accepted Answer

It keeps on the order of length squared pairs. Dropping the future half does not make the count linear.

Question 7

What does a materialized logit tensor grow with?

Accepted Answer

Batch, sequence length, and vocabulary. It does not grow with the head count.

Question 8

What happens to Adam moment storage if the sequence gets longer and the parameters stay the same?

Accepted Answer

It stays the same size. Moments are stored per parameter, not per token position.

Question 9

What shape is a parameter's gradient tensor?

Accepted Answer

The same shape as that parameter. Sequence length does not add an axis to it.

Question 10

What does the KV cache store, and which factors set its bytes?

Accepted Answer

Keys and values. The factor of two is K and V. Bytes grow with layers, KV heads, head dimension, sequence length, and bytes per element.

Question 11

During single-token decode, what is the shape of one head's new scores?

Accepted Answer

One query by the cache length. It is a row, not a square over the cache.

Question 12

Where is the heavy attention work in prefill, and where is it in decode?

Accepted Answer

Prefill over a prompt is heavy on the score computation. Decode is heavy on reading the KV cache.

Question 13

Why does paged attention store the KV cache in blocks?

Accepted Answer

So each sequence does not need one contiguous allocation.

Question 14

A paged KV cache holds a fixed number of tokens per block. How many blocks does one sequence use?

Accepted Answer

The ceiling of its token count divided by the block size.

Question 15

How do a rectangular batched cache and a paged cache treat padding?

Accepted Answer

A rectangular cache reserves positions out to the padded length. A paged cache allocates blocks for tokens that were actually stored.

Question 16

What does FlashAttention change, and what architecture does it leave in place?

Accepted Answer

It tiles the exact attention computation so the full score matrix is not materialized. It is not a different model architecture.

Question 17

How does the FlashAttention backward pass obtain the scores it needs?

Accepted Answer

It recomputes the tiled attention from Q, K, and V instead of reading a stored score matrix.

Question 18

What is the memory difference between a causal flag inside the kernel and a dense additive mask?

Accepted Answer

A causal flag lets the kernel skip future tiles without an S by S tensor. A dense additive mask of that shape is quadratic memory of its own.

Question 19

How many bytes does one element use in bf16, fp16, and fp32?

Accepted Answer

bf16 and fp16 use 2 bytes. fp32 uses 4.

Question 20

How do bf16 and fp16 differ in range and mantissa, and which one more often needs loss scaling?

Accepted Answer

bf16 keeps a wide exponent range like fp32 and has fewer mantissa bits. fp16 has a smaller range, so fp16 training more often uses loss scaling.

Question 21

Why must a large negative attention mask be chosen carefully when the scores are in fp16?

Accepted Answer

fp16's finite range tops out near 65504, so a huge mask such as -1e9 saturates. The mask has to fit in fp16 or be applied in wider math.

Question 22

Why do softmax implementations subtract the maximum score before the exponential?

Accepted Answer

The shift leaves the softmax unchanged and keeps the exponential from overflowing.

Question 23

What scale is applied to attention scores before the softmax, and what is it for?

Accepted Answer

Divide by the square root of the head dimension, so the dot product does not spread more as that width grows.

Question 24

What should the token cross-entropy consume?

Accepted Answer

Raw logits and integer target ids. A softmax applied before that loss squashes the distribution a second time.

Question 25

Why is Adam's stored state larger than the weights alone?

Accepted Answer

Adam stores a first moment and a second moment for every parameter.

Question 26

What extra weight copy does mixed-precision training commonly keep?

Accepted Answer

An fp32 master copy of the weights. The matmul reads a lower-precision cast of that master.

Question 27

What is lost if a mixed-precision run saves only the bf16 weights?

Accepted Answer

The extra bits in the fp32 master. The bf16 file is a rounding of that master.

Question 28

What extra state does dynamic loss scaling add to an exact fp16 resume?

Accepted Answer

The scaler state, including the current scale. The weights do not imply that scale.

Question 29

What does gradient checkpointing recompute, and what does it leave unchanged?

Accepted Answer

It recomputes activations during the backward pass to save memory. It does not reduce the parameter count.

Question 30

How is activation checkpointing different from a checkpoint file?

Accepted Answer

Activation checkpointing recomputes tensors during the backward pass. A checkpoint file is saved weights and other state for a later load.

Question 31

Does a longer sequence make the saved weight file larger?

Accepted Answer

No. Weight-file bytes follow the parameter count and the dtype. Sequence length changes activations, not those parameters.

Question 32

What does each of these parallel splits cut?

Accepted Answer

Data parallel copies the full weights and splits the batch. Tensor parallel shards a matrix multiply. Pipeline parallel shards layers. Sequence or context parallel shards the sequence.

Question 33

When the only splits are data parallel, tensor parallel, and pipeline parallel, what is the data-parallel degree?

Accepted Answer

The GPU count divided by the tensor-parallel degree and the pipeline degree. It is not the raw GPU count.

Question 34

When do data parallel and tensor parallel move data?

Accepted Answer

Data parallel all-reduces gradients once per optimizer step. Tensor parallel communicates inside the layer.

Question 35

What keeps data-parallel replicas on the same weights?

Accepted Answer

An all-reduce of the gradients before the optimizer step. Skipping it lets the replicas diverge.

Question 36

What crosses a pipeline-stage boundary during a training step?

Accepted Answer

Hidden states, and their gradients on the way back. Stages do not send the full weight set each step.

Question 37

What happens in a pipeline step that has only one microbatch?

Accepted Answer

Other stages sit idle while that microbatch is on a different stage.

Question 38

What does each ZeRO stage shard?

Accepted Answer

Stage 1 shards optimizer state. Stage 2 also shards gradients. Stage 3 also shards parameters.

Question 39

Which memory does ZeRO leave on the rank that runs the forward?

Accepted Answer

The local activation tensor. ZeRO shards optimizer state and, at later stages, gradients and parameters.

Question 40

When ZeRO stage 3 runs a layer, where are that layer's full parameters?

Accepted Answer

They are gathered for the layer that is about to run. The rank does not keep every full parameter for the whole step.

Question 41

What does offloading Adam's moments to the CPU change?

Accepted Answer

Where the moments sit. They are still stored and still updated. Their count does not disappear.

Question 42

What is the global batch, counted in sequences?

Accepted Answer

Microbatch times accumulation steps times the data-parallel degree.

Question 43

Which batch size sets activation memory on one device?

Accepted Answer

The microbatch on that device. The global batch also counts sequences on other steps or other replicas.

Question 44

What activation memory does sequential gradient accumulation hold at one time?

Accepted Answer

About one microbatch. Earlier microbatches keep their gradients and release their activations.

Question 45

Equal microbatches each produce a mean gradient, and those gradients are summed over N accumulation steps. What does the update need?

Accepted Answer

A division by N. Otherwise the summed gradient is N times the mean over the combined examples.

Question 46

When does gradient accumulation step the optimizer, and what does the learning-rate schedule count?

Accepted Answer

It adds gradients across microbatches and steps the optimizer once. The schedule counts optimizer steps.

Question 47

What does a larger batch change about an optimizer step, and which width does it leave alone?

Accepted Answer

It changes how many sequences share that step. It does not change d_model.

Question 48

Which tokenizer can serve a finished set of weights?

Accepted Answer

The tokenizer the weights were trained with. A different id map makes the same weights read different text.

Question 49

Which tokenizer must evaluation use when it reports a token-level loss?

Accepted Answer

The same tokenizer as training. That loss is a loss on those ids.

Question 50

Why are token-level losses from two tokenizers not the same unit?

Accepted Answer

Each is an average per token, and the tokenizers can split the same text into different numbers of tokens.

Question 51

Can a chat template change the ids when the weights and the vocab are unchanged?

Accepted Answer

Yes. The template wraps the text before tokenization, so the model sees a different id sequence.

Question 52

What must a BPE tokenizer artifact include besides the vocab list?

Accepted Answer

The merge list. Without those merges, the vocab alone does not define the encoding.

Question 53

Where do the RoPE base and the maximum position come from at load?

Accepted Answer

From the config. The weight file does not invent them, and a different position config changes the positions.

Question 54

How are standard RoPE frequencies obtained?

Accepted Answer

They are computed from the base and the head dimension. They are not a vocabulary-sized table in the weight file.

Question 55

Which attention vectors does RoPE rotate?

Accepted Answer

Queries and keys. Values are left unrotated.

Question 56

How does a learned absolute position table differ from RoPE inside the weight file?

Accepted Answer

The learned table has one row per maximum position and width d_model. Standard RoPE has no position-row parameter.

Question 57

Which architecture fields have to be known before weight tensors can be placed into a module?

Accepted Answer

Layer count, hidden size, head count, intermediate size, and vocab size.

Question 58

What does the tied-embedding flag tell a loader?

Accepted Answer

Whether the output projection is the input embedding. The wrong flag looks for a matrix the file does not store, or ignores the tie.

Question 59

How does loading safetensors differ from loading a pickle checkpoint?

Accepted Answer

safetensors reads named tensors and does not run pickled code. A pickle load can execute code.

Question 60

What does a sharded weight directory need besides the shard files themselves?

Accepted Answer

An index from each tensor name to the shard that holds it. One shard is not the whole model.

Question 61

What metadata does a tensor-parallel weight shard need so it can be loaded?

Accepted Answer

The parallel degree and the axis it was cut on. A shard saved at a different degree does not match the module.

Question 62

What does one pipeline-parallel checkpoint contain?

Accepted Answer

That stage's layers. The full stack is the stage files reunited in layer order.

Question 63

Can one rank's ZeRO stage-3 file be loaded as a normal full model?

Accepted Answer

No. That file is a partition. The partitions have to be consolidated first.

Question 64

What must a checkpoint contain to load the model, to resume training, and to resume exactly?

Accepted Answer

A weights-only checkpoint can load the model. Resuming also needs optimizer state and the step count. An exact resume also needs RNG state, and some kernels can still be nondeterministic.