Transformer basics

Does a transformer layer change the shape of its input?

No. Each layer maps batch × length × d_model to the same shape, which is why layers can be stacked.

What does the embedding layer do with a token ID?

Returns one row of the embedding matrix, which has one row per vocabulary entry and d_model columns.

With absolute position vectors, what do two copies of the same token at different indexes share?

The same token embedding. Their inputs differ only by the added position vector.

What happens to self-attention with no position information?

It treats the input as a set: reordering the tokens just reorders the outputs.

What is relative position encoding?

Position information that depends on the distance between two tokens instead of each token's absolute index.

Where is RoPE applied?

To the query and key vectors inside each attention head, not to the values or the token embeddings.

How are Q, K, and V computed from the hidden states?

Three learned matrices multiply the same hidden states, giving one query, key, and value vector per token.

What is the shape of one head's attention score matrix?

Query length × key length. For self-attention on n tokens it is n × n.

In scaled dot-product attention, what is the output for one query?

A weighted average of the value vectors, using that query's softmax weights.

Do two queries in the same head share one set of attention weights?

No. Each query has its own row of scores and its own softmax.

Over which axis is the attention softmax taken?

Over the keys, so each query's weights across positions sum to one.

A language model applies softmax in two places. What is each over?

In attention, over key positions. At the output, over the vocabulary.

With d_model 768 and 12 heads, what is each head's width?

64. The model width is split evenly across the heads.

What happens to the head outputs?

They are concatenated back to width d_model and passed through the output projection.

Why use several heads instead of one wide head?

Each head has its own projections, so different heads can attend to different patterns at the same time.

What does a causal mask prevent?

Any position attending to later positions, so a prediction cannot see the token it is predicting.

How is a mask applied to attention scores?

Blocked positions get a very large negative value before softmax, so their weight becomes effectively zero.

Why is a padding mask needed in a batch?

Shorter sequences are padded to a common length. The mask stops real tokens from attending to the padding.

Under a causal mask, what can the first position attend to?

Only itself.

What is the shape of the original feed-forward sublayer?

Two linear layers: d_model up to d_ff, usually 4 × d_model, a nonlinearity, then back down to d_model.

Does the feed-forward block mix information across positions?

No. It is applied to each position separately, with the same weights. Only attention mixes positions.

Which nonlinearities are common in transformer feed-forward blocks?

ReLU in the original paper, then GELU, and now gated forms such as SwiGLU.

What is a gated feed-forward block such as SwiGLU?

Two up-projections, one passed through an activation and multiplied elementwise with the other, then a down-projection.

What does a residual connection add?

The sublayer's input to its output, so each sublayer writes an update into a running stream.

Why do deep transformers need residual connections?

They give gradients a direct path through the stack, which keeps very deep models trainable.

Over what does layer normalization compute its statistics?

Over the d_model features of each token separately, not across the batch or the sequence.

Which learned parameters does layer normalization have?

A scale and a shift per feature. RMSNorm keeps only the scale.

In a pre-norm block, what order are the operations in?

Normalize, apply attention or feed-forward, then add the result to the residual stream.

What usually comes after the last pre-norm layer?

A final normalization before the output projection.

What is the shape of the output logits?

Batch × sequence length × vocabulary size: one score per vocabulary entry per position.

Which logits row predicts the next token after a prompt?

The row for the last prompt position.

How are labels aligned in next-token training?

Shifted by one: the label at position i is the input token at position i + 1.

Why can a decoder train on all positions in parallel?

The causal mask lets every position compute its prediction from earlier tokens in a single forward pass.

Why is generation sequential?

Each new token depends on the previous one, so tokens are produced one forward step at a time.

Where is dropout applied in a transformer?

Commonly on attention weights, on sublayer outputs before the residual add, and on embeddings.

Is a dropout mask part of the saved model?

No. Masks are sampled fresh during training and are not stored.

With dropout off and weights fixed, what is the forward pass a function of?

Only the input tokens, apart from small numeric differences from hardware or kernel choices.

Is greedy decoding deterministic?

Yes, in principle: it always takes the highest-scoring token. Sampling with temperature above zero is not.

What does temperature do to the output distribution?

Divides logits before softmax. Below 1 sharpens the distribution, above 1 flattens it.

In an encoder-decoder model, where does cross-attention get its keys and values?

From the encoder's output. The queries come from the decoder.

Why are encoder-only models good for classification and embeddings?

Every token sees the whole input in both directions, giving rich representations of the full text.

What is a CLS token?

A special token added to the input whose final state is used to represent the whole sequence, for example in classification.

Roughly how many parameters does one standard layer have?

About 12 × d_model²: about 4 × d_model² in attention and 8 × d_model² in a 4× feed-forward block.

How does attention cost grow with sequence length?

Quadratically, since every query is scored against every key.

What does the KV cache store?

The keys and values of earlier positions for every layer, so each new token computes only its own.

What sets the context window limit?

The longest sequence the model was trained to handle well, and the memory available for attention and the KV cache.

How is a pretrained encoder adapted for classification?

Add a small output layer on the pooled representation and fine-tune, sometimes freezing most of the encoder.

What is feature extraction with a frozen transformer?

Keep the model's weights fixed and use its hidden states as input features for a separate, lighter model.

What does LoRA add to a transformer?

Small low-rank matrices next to chosen weight matrices. Only these are trained, and the original weights stay frozen.