Transformer basics

Question 1

What does a token embedding do with a token id?

Accepted Answer

It maps the id to a vector whose width is d_model.

Question 2

What is the shape of the token-embedding matrix?

Accepted Answer

Vocabulary size by d_model.

Question 3

Two positions contain the same token id. Before any position signal is applied, how do their token embeddings compare?

Accepted Answer

They are the same vector, because both ids read the same row of the embedding table.

Question 4

How does the original transformer combine the token embedding with positional information?

Accepted Answer

It adds a position vector of width d_model to the token embedding.

Question 5

What does the original sinusoidal positional encoding write into the position vector?

Accepted Answer

Sine and cosine of the position, using a different frequency for each pair of coordinates.

Question 6

In the original sinusoidal encoding, which coordinates are sine and which are cosine?

Accepted Answer

With zero-based indexes, coordinate 2i is sine and coordinate 2i+1 is cosine of the same frequency.

Question 7

In the original sinusoidal encoding, what happens to the oscillation as the dimension index i increases?

Accepted Answer

The divisor is 10000^(2i/d_model), so a larger i oscillates more slowly.

Question 8

In the original transformer, what scale is applied to the token embeddings before the position vector is added?

Accepted Answer

They are multiplied by the square root of d_model.

Question 9

How does rotary position encoding inject position?

Accepted Answer

It rotates query and key coordinate pairs by an angle that depends on position. It is not a vector added to the token embedding.

Question 10

Which of the query, key, and value does RoPE rotate?

Accepted Answer

The query and the key. It does not rotate the value.

Question 11

What does a RoPE rotation do to the length of a query or a key?

Accepted Answer

It preserves the length. The rotation changes direction and does not rescale the vector.

Question 12

After RoPE, what does the dot product of a query at position m and a key at position n depend on?

Accepted Answer

It depends on the two vectors and on the relative offset between m and n.

Question 13

How does ALiBi represent position?

Accepted Answer

It adds a distance bias to the attention scores. It does not add a position vector to the token embedding.

Question 14

Where is the ALiBi distance bias inserted?

Accepted Answer

It is added to the attention scores before the softmax.

Question 15

How do ALiBi slopes differ across heads?

Accepted Answer

Each head has its own fixed slope, so some heads penalize distance more steeply than others.

Question 16

Where does a relative attention bias enter the block?

Accepted Answer

It is added to the score. It is not added to the value.

Question 17

What limits the indexes a learned absolute position table can represent?

Accepted Answer

The table has one stored row per index up to a fixed maximum. An index past that maximum has no row.

Question 18

Where are absolute position vectors applied, compared with RoPE and ALiBi?

Accepted Answer

Absolute sinusoidal or learned positions are added at the input embedding. RoPE and ALiBi are applied inside attention.

Question 19

With no position signal and no mask that depends on position, what symmetry does self-attention have?

Accepted Answer

It is permutation equivariant: reorder the tokens, and the outputs reorder the same way.

Question 20

Self-attention has no position signal and no mask that depends on position. If the token at a fixed index is swapped with another token, does the output vector at that index stay the same?

Accepted Answer

No. The output moves with the token, so the vector left at that index belongs to the token that arrived there.

Question 21

How are the query, key, and value vectors produced from the attention sublayer input?

Accepted Answer

By learned linear projections.

Question 22

In self-attention, are the query, key, and value the same vector?

Accepted Answer

No. They come from the same sequence through three different learned projections.

Question 23

Inside the attention sublayer, is a pointwise activation such as ReLU applied between the input and Q, K, and V?

Accepted Answer

No. Those three maps are linear. The softmax is applied later, to the scores.

Question 24

In self-attention, where do the query, key, and value come from?

Accepted Answer

All three come from the same sequence.

Question 25

In cross-attention, where do the query, key, and value come from?

Accepted Answer

The query comes from the decoder side. The key and value come from the encoder sequence.

Question 26

Which stack contains cross-attention in a standard encoder-decoder transformer?

Accepted Answer

The decoder. The encoder uses self-attention and does not cross-attend.

Question 27

How do the score-matrix shapes of self-attention and cross-attention differ?

Accepted Answer

Self-attention scores are square, sequence by sequence. Cross-attention scores have shape decoder length by encoder length.

Question 28

In scaled dot-product attention, the query-key products are divided by which quantity before the softmax?

Accepted Answer

The square root of d_k.

Question 29

When heads split a fixed d_model, which width does the attention scale use?

Accepted Answer

d_k, the key width of one head, which is narrower than d_model.

Question 30

Why are query-key products scaled before the softmax?

Accepted Answer

Raw dot products grow with dimension, and large scores saturate the softmax.

Question 31

Assume query and key components are independent, with mean 0 and variance 1. Why is the divisor the square root of d_k?

Accepted Answer

The dot product then has variance d_k, so its standard deviation is the square root of d_k.

Question 32

In the original scaled dot-product attention, is 1/sqrt(d_k) a learned parameter?

Accepted Answer

No. It is the constant fixed by the head width.

Question 33

For one query, what is true of the attention weights after the softmax?

Accepted Answer

They are nonnegative, and they sum to 1 over the keys.

Question 34

Do the queries in one head share a single pool of softmax mass?

Accepted Answer

No. Each query has its own distribution over the keys.

Question 35

Can the weights on one key, summed across queries, exceed 1?

Accepted Answer

Yes. Each query has its own mass, so those weights can add up to more than 1.

Question 36

Must the attention weight from position i to position j equal the weight from j to i?

Accepted Answer

No. The two directions are separate softmaxes of different query-key scores.

Question 37

What is one head's output before the output projection mixes heads?

Accepted Answer

A weighted sum of the value vectors. The scores set the weights, and the values are what get mixed.

Question 38

Before the output projection, one head's output is a weighted sum of value vectors. Why is that sum a convex combination?

Accepted Answer

The weights are nonnegative and sum to 1, so the result lies in the convex hull of the value vectors.

Question 39

A query-key score is negative, and it is not masked. What is true of that key's attention weight?

Accepted Answer

The weight is still nonnegative. A negative score is not a negative weight.

Question 40

Is a raw query-key score already a probability?

Accepted Answer

No. The probability is the softmax of the scaled scores. A raw score can be negative or larger than 1.

Question 41

What happens to one query's weights if the same finite constant is added to every one of its scores?

Accepted Answer

The weights stay the same.

Question 42

How does a causal mask stop position i from attending to a later position j greater than i?

Accepted Answer

It sets those scores to negative infinity before the softmax, which makes the corresponding weights 0.

Question 43

Under a causal mask, may position i attend to itself?

Accepted Answer

Yes. The mask blocks keys with index greater than i, and the key at i stays allowed.

Question 44

What happens if a forbidden key is given score 0, rather than negative infinity, before the softmax?

Accepted Answer

It still receives weight. Zero is an ordinary finite score.

Question 45

In a causal attention head, what is the mix at the first position?

Accepted Answer

The weight on itself is 1, and the mix equals that position's value vector.

Question 46

What does a padding mask do?

Accepted Answer

It stops attention from landing on pad positions.

Question 47

How is a padding mask different from a masked-language-model mask token?

Accepted Answer

A padding mask blocks scores at pad keys. A mask token is an input symbol that stands in for a hidden token.

Question 48

What can a position see in encoder self-attention?

Accepted Answer

It is bidirectional, so the position may attend to tokens on both sides.

Question 49

What can a position see in decoder self-attention?

Accepted Answer

It is causal, so position i cannot attend to a position j greater than i.

Question 50

Does the causal mask hide encoder keys from a decoder query in standard cross-attention?

Accepted Answer

No. A decoder query may use the encoder sequence. The causal mask constrains decoder self-attention.

Question 51

In the original transformer, what does the decoder's masked multi-head attention mask out?

Accepted Answer

Future target positions, by putting negative infinity on those scores. It is a causal mask.

Question 52

What does a decoder-only stack leave out, compared with an encoder-decoder?

Accepted Answer

It has no encoder and no cross-attention.

Question 53

Does decoder-only mean the model has no attention?

Accepted Answer

No. It still has causal self-attention. It lacks an encoder and cross-attention.

Question 54

Does an encoder-only model use a causal mask for the usual masked-token objective?

Accepted Answer

No. That objective predicts hidden tokens from both sides, so self-attention is not causal.

Question 55

Which sublayers make one original encoder layer?

Accepted Answer

Bidirectional self-attention and a position-wise feed-forward block.

Question 56

Which sublayers make one original decoder layer?

Accepted Answer

Causal self-attention, cross-attention over the encoder, and a position-wise feed-forward block.

Question 57

Which sublayers make one decoder-only layer?

Accepted Answer

Causal self-attention and a position-wise feed-forward block.

Question 58

Why can a later causal layer not read a future token that earlier layers were forbidden to use?

Accepted Answer

Every self-attention layer is causal, and the feed-forward block does not look at other positions.

Question 59

In a causal stack, which tokens can affect the hidden state at position i?

Accepted Answer

Tokens at positions less than or equal to i. Later tokens cannot.

Question 60

A causal prompt is run in one parallel forward pass. Which prompt tokens can position i use?

Accepted Answer

Itself and earlier prompt tokens. A parallel pass still applies the causal mask.

Question 61

What does a residual connection add?

Accepted Answer

The sublayer input and the sublayer output.

Question 62

Why do attention and the feed-forward block both return vectors of width d_model?

Accepted Answer

Those outputs are added to a residual of width d_model. The wide feed-forward activation is projected back before the add.

Question 63

Does the residual add by itself move a vector from one position to another?

Accepted Answer

No. It adds a vector onto the same position. Mixing across positions is done by attention.

Question 64

What is post-norm?

Accepted Answer

LayerNorm of the sum of the sublayer input and the sublayer output.