Transformer basics

How many bytes does one element use in fp32, bf16, int8, and int4?

4, 2, 1, and 0.5 bytes.

Does a longer sequence make the saved weight file larger?

No. Weights do not depend on sequence length. Longer sequences only need more activation and KV cache memory at run time.

What takes GPU memory during training besides the weights?

Gradients, optimizer state, and activations saved for the backward pass.

How much memory does Adam add per parameter in mixed precision?

Typically 12 bytes: an fp32 master copy plus two fp32 moments, on top of the 16-bit weights and gradients.

Which parameters receive Adam moments?

Every trainable parameter. Frozen parameters, such as the base model under LoRA, need none.

What makes activation memory grow?

Batch size, sequence length, model width, and number of layers. Attention scores add a term that grows with length squared.

How does one head's attention score matrix grow?

With the square of sequence length, multiplied by batch size and number of heads if all are materialized.

What does FlashAttention change, and what does it leave the same?

It computes attention in tiles without storing the full score matrix, saving memory and time. The model and its outputs stay the same.

What is activation checkpointing?

Saving only some activations in the forward pass and recomputing the rest during backward, trading extra compute for less memory.

What is gradient accumulation?

Summing gradients over several micro-batches before one optimizer step, to reach a large effective batch on limited memory.

With gradient accumulation, when does the optimizer step, and what does the LR schedule count?

Once per accumulated batch. The schedule counts optimizer steps, not micro-batches.

What is the global batch size?

Micro-batch size × accumulation steps × number of data-parallel replicas.

When is dropout applied?

In training mode only. Serving runs in eval mode with dropout off.

Why is bf16 preferred to fp16 for training?

It has the same exponent range as fp32, so it rarely overflows and does not need loss scaling.

What is loss scaling in fp16 training?

Multiplying the loss before backward so small gradients do not underflow, then dividing the gradients back before the update.

What are common causes of NaN loss?

Overflow in fp16, a learning rate that is too high, bad input data, or division by zero in a custom op.

What is data parallelism?

Each GPU holds a full model copy, processes different data, and gradients are averaged across GPUs each step.

What do ZeRO and FSDP do?

Shard optimizer state, gradients, and optionally weights across data-parallel GPUs, so each GPU stores only part.

What is tensor parallelism?

Splitting individual weight matrices across GPUs, so each layer's math is shared. It needs fast links between those GPUs.

What is pipeline parallelism?

Putting different layers on different GPUs and passing micro-batches through them in stages.

What is a pipeline bubble?

Idle time when some stages wait for work at the start and end of each batch. More micro-batches shrink it.

What metadata does a tensor-parallel weight shard need to load correctly?

Which tensor it belongs to, which dimension was split, its index among the shards, and the total shard count.

What is all-reduce?

A collective operation where every GPU ends up with the sum or average of all GPUs' values, used for gradients.

What is NCCL?

NVIDIA's library for collective communication between GPUs, used by most distributed training frameworks.

What must a training checkpoint contain to resume exactly?

Weights, optimizer state, LR scheduler state, step count, random number generator states, and data loader position.

What is lost if only the weights are saved?

Optimizer moments, schedule position, and data position, so a resume will not match the original run.

Why use the safetensors format?

It stores raw tensors with a simple header, loads fast with memory mapping, and cannot run code on load the way pickle files can.

Why must the tokenizer be versioned with the weights?

Token IDs only mean something with the exact vocabulary used in training. A mismatched tokenizer gives garbage outputs.

What does the model config file record?

Architecture settings such as layer count, width, heads, vocabulary size, and position scheme, needed to build the model before loading weights.

What are the prefill and decode phases?

Prefill processes the whole prompt at once and is compute-bound. Decode generates one token per step and is memory-bandwidth bound.

How do you estimate KV cache size?

2 × layers × KV heads × head dim × sequence length × batch size × bytes per element.

How does grouped-query attention reduce serving memory?

Several query heads share one key and value head, so the KV cache shrinks by that grouping factor.

What is paged attention?

Storing the KV cache in fixed-size blocks allocated on demand, like virtual memory pages, which reduces wasted memory.

What is continuous batching?

Adding new requests to the running batch as others finish, instead of waiting for the whole batch to end.

What does chunked prefill do with a long prompt?

Processes it in pieces across several steps, mixed with other requests' decode steps, so one long prompt does not stall the batch.

What is time to first token?

The time from receiving a request to returning the first output token, mostly queueing plus prefill.

What is time per output token?

The average time between generated tokens during decode.

What is the throughput and latency trade-off in serving?

Larger batches raise tokens per second overall, but each request may wait longer.

What is weight quantization?

Storing weights in fewer bits, such as int8 or int4, to cut memory and bandwidth, with some loss of accuracy.

Why re-run evaluations after quantizing?

Accuracy loss varies by model and task, so the quantized artifact must be checked before it replaces the original.

What is KV cache quantization?

Storing cached keys and values in lower precision to fit longer contexts or larger batches.

What is speculative decoding?

A small draft model proposes several tokens and the large model checks them in one pass, keeping only those it agrees with.

Does speculative decoding change the output distribution?

No, with the standard acceptance rule. It only changes speed.

What does merging a LoRA adapter do?

Adds the low-rank update into the base weights, giving one weight file with no adapter overhead at serving time.

What is multi-LoRA serving?

One base model in memory serving many small adapters, chosen per request.

Why can the same input give slightly different outputs on a GPU?

Floating-point sums in different orders, from batching, kernel choice, or parallelism, give tiny differences that can change a sampled token.

What limits the maximum sequence length a server accepts?

The model's trained context length and the memory left for the KV cache after loading weights.

What are first steps for a GPU out-of-memory error in training?

Lower the micro-batch, raise accumulation, turn on activation checkpointing, or shard state with FSDP or ZeRO.

What is model FLOPs utilization (MFU)?

Achieved training FLOPs per second divided by the hardware's peak. It shows how efficiently the GPUs are used.