Transformer basics

What does a tokenizer do?

Splits text into tokens and maps them to IDs, and maps IDs back to text. Different models have different tokenizers.

What is byte-pair encoding (BPE)?

A subword method that starts from characters or bytes and repeatedly merges the most frequent pairs, so common words are single tokens and rare words are split.

What is the vocabulary size?

The number of distinct tokens the model knows. It sets the size of the embedding table and of the output layer.

What does the token embedding do?

Looks up a learned vector of width d_model for each token ID.

What is d_model?

The width of the vector that represents each token as it moves through the model's layers.

Why does a transformer need position information?

Attention by itself ignores order, treating inputs as a set. Position signals let it tell 'dog bites man' from 'man bites dog'.

What is sinusoidal positional encoding?

Fixed sine and cosine values at different frequencies, added to each position's embedding in the original transformer.

What is a learned absolute position embedding?

A trained vector for each position index, added to the token embedding. It cannot represent positions beyond the table's length.

What is rotary position embedding (RoPE)?

Rotating query and key vectors by an angle that depends on position, so their dot product depends on the distance between tokens.

What is ALiBi?

Adding a penalty to attention scores that grows with the distance between tokens, instead of adding position vectors.

What are queries, keys, and values?

Three vectors made from each token by separate learned projections. A query is compared with keys to get weights, and the weights mix the values.

What is scaled dot-product attention?

softmax(QKᵀ / √d_k) V. Scores are query-key dot products divided by the square root of the key width, turned into weights by softmax.

Why divide attention scores by √d_k?

Dot products grow with vector width. Scaling keeps the softmax from saturating, so gradients stay useful.

What is true of one query's attention weights?

They are non-negative and sum to 1 across the keys it can see.

What is self-attention?

Attention where queries, keys, and values all come from the same sequence, so each token gathers information from the others.

What is cross-attention?

Attention where queries come from one sequence, such as the decoder, and keys and values come from another, such as the encoder output.

What is multi-head attention?

Several attention operations run in parallel on smaller slices of the width, each with its own projections. Their outputs are joined and projected back.

How does the head count affect each head's width?

With d_model fixed and split evenly, each head's width is d_model divided by the number of heads.

What does the attention output projection do?

Mixes the joined outputs of all heads back into a d_model-wide vector for each token.

What is a causal mask?

A mask that stops each position from attending to later positions, by setting those scores to negative infinity before the softmax.

What is a padding mask?

A mask that stops attention to padding tokens added to make sequences in a batch the same length.

What is the feed-forward block in a transformer layer?

A small two-layer network applied to each token separately, usually widening to about 4 × d_model and back, with a nonlinearity in between.

What is the residual stream?

The running vector for each token that every sublayer reads from and adds its output back into.

What does layer normalization do in a transformer?

Rescales each token's vector to a standard mean and spread, with a learned scale and shift. It does not depend on the batch.

What is the difference between pre-norm and post-norm?

Pre-norm normalizes the input to each sublayer before it runs, leaving the residual path untouched. Post-norm normalizes after adding the residual. Pre-norm is more stable for deep models.

What is RMSNorm?

A simpler normalization that divides by the root mean square of the vector without subtracting the mean. Many recent language models use it.

What makes up one decoder-only transformer layer?

Masked self-attention and a feed-forward block, each with normalization and a residual connection.

What is an encoder-only model?

A transformer whose attention sees the whole input in both directions, such as BERT. It suits classification and embeddings.

What is a decoder-only model?

A transformer with causal self-attention that predicts the next token, such as GPT-style models. It still uses attention.

What is an encoder-decoder transformer?

An encoder reads the input with full attention, and a decoder generates output with causal self-attention plus cross-attention to the encoder.

What is masked language modeling?

A training task that hides some input tokens and asks the model to predict them from both sides. It is used for encoder models.

What is next-token prediction?

A training task where each position predicts the token that follows it. With a causal mask, every position is trained in one pass.

In next-token training, what is the label for position i?

The token at position i + 1.

What are logits in a language model?

The raw scores over the whole vocabulary at a position, before softmax turns them into probabilities.

What does the output layer (unembedding) do?

Projects the final hidden vector at a position to one score per vocabulary token.

What is weight tying?

Using the same matrix for the input embedding and the output layer, which saves parameters.

After processing a prompt, which position gives the next token's distribution?

The last position.

What is greedy decoding?

Always picking the single most likely next token.

What does temperature do when sampling?

Divides logits before the softmax. Below 1 sharpens the distribution toward likely tokens; above 1 flattens it.

What are top-k and top-p sampling?

Top-k samples from the k most likely tokens. Top-p samples from the smallest set whose probabilities add up to p.

What is the KV cache?

Stored keys and values for tokens already processed, so each new token only computes its own and attends to the stored ones.

How does KV cache size grow?

Linearly with sequence length, and also with the number of layers, heads, and batch size.

Which tensor grows quadratically with sequence length?

The attention score matrix, which is sequence length by sequence length for each head.

What is the context window?

The maximum number of tokens a model can take in one call, counting the prompt and the generated output.

What are multi-query and grouped-query attention?

Variants where many query heads share one or a few sets of key and value heads. They shrink the KV cache and speed up generation.

What is FlashAttention?

An exact attention algorithm that computes in blocks to avoid storing the full score matrix in slow memory, making attention faster and leaner.

In a standard layer with a 4× feed-forward block, where do most parameters sit?

In the feed-forward block, about 8 × d_model² versus about 4 × d_model² for attention, ignoring biases.

What is a mixture-of-experts layer?

A layer with many feed-forward experts and a router that sends each token to a few of them, adding parameters without adding much compute per token.

What do scaling laws describe?

How a model's loss falls in a predictable way as parameters, training data, and compute grow.