Transformer basics

Question 1

In scaled dot-product attention, what is the vector produced for one query?

Accepted Answer

A weighted sum of the value vectors, with weights from the softmax of the scaled query-key scores.

Question 2

Before the softmax, what is the score between one query and one key?

Accepted Answer

Their dot product, divided by the square root of the key dimension.

Question 3

In one head of self-attention, what does the matrix of query-key products contain?

Accepted Answer

One score for every query position paired with every key position.

Question 4

Which pair of vectors sets how much one position attends to another?

Accepted Answer

The query at the receiving position and the key at the source position.

Question 5

In the attention formula, what is d_k?

Accepted Answer

The width of each query vector and each key vector in that head.

Question 6

After the attention weights multiply the values, what is the shape of the result?

Accepted Answer

One vector per query, with the same width as each value vector.

Question 7

In self-attention, where do queries, keys, and values come from?

Accepted Answer

All three are projections of the same sequence of hidden states.

Question 8

Why does a full attention score matrix grow quickly as the sequence gets longer?

Accepted Answer

Every query is scored against every key, so the number of scores grows with the square of the length.

Question 9

How does a hidden vector become a query, a key, and a value?

Accepted Answer

Three learned linear maps, with separate weights, project that vector.

Question 10

Are the query, key, and value projection weights different at each position?

Accepted Answer

No. The same weights are applied independently at every position.

Question 11

What job does each of query, key, and value do?

Accepted Answer

The query searches, the key is what gets matched, and the value is the content added when it matches.

Question 12

Which widths must match for a query-key score to be defined?

Accepted Answer

Query and key must share a width. The value may use a different width.

Question 13

In cross-attention, which stream is projected into queries, and which into keys and values?

Accepted Answer

The decoder stream supplies queries. The encoder stream supplies keys and values.

Question 14

After attention heads produce their outputs, what returns the result to the model width?

Accepted Answer

The head outputs are concatenated and then mixed by a learned output map.

Question 15

For one query, the attention softmax is a distribution over what?

Accepted Answer

The key positions, so the weights on the sources sum to one.

Question 16

Do two queries in the same head share a single set of attention weights?

Accepted Answer

No. Each query has its own softmax over the keys, and each of those rows sums to one.

Question 17

Does the attention softmax pool probability across heads?

Accepted Answer

No. Each head has its own scores and its own softmax.

Question 18

A score was set to negative infinity before the attention softmax. What is its weight afterward?

Accepted Answer

Zero, so the corresponding value does not enter the weighted sum.

Question 19

What does the attention softmax force to sum to one?

Accepted Answer

The attention weights over key positions for that query.

Question 20

Assume query and key entries are independent, with mean zero and variance one. What is the variance of their dot product?

Accepted Answer

d_k, the count of independent unit-variance terms inside the dot product.

Question 21

Why are query-key scores divided by sqrt(d_k)?

Accepted Answer

To hold score variance near one as d_k grows, so the softmax does not saturate.

Question 22

What happens if the 1/sqrt(d_k) factor is removed and d_k is large?

Accepted Answer

Scores become large, the softmax becomes nearly one-hot, and its gradient becomes tiny.

Question 23

Why divide by the square root of d_k rather than by d_k itself?

Accepted Answer

The standard deviation of the dot product grows with the square root of the number of summed terms.

Question 24

When several heads split the model width, which length is used in the attention scale?

Accepted Answer

The key dimension of that head, not the full model width.

Question 25

What does multi-head attention run inside a single sublayer?

Accepted Answer

Several attentions side by side, each with its own projections and its own softmax.

Question 26

What does a layer gain from several heads instead of one attention pattern?

Accepted Answer

Several different distributions over positions, combined in that same layer.

Question 27

Which procedure is multi-head attention?

Accepted Answer

Softmax each head separately, concatenate the outputs, and apply a linear map.

Question 28

If the model width stays fixed, what does the usual head split do when you add heads?

Accepted Answer

It makes each head narrower, because head count times head width equals the model width.

Question 29

Heads and extra blocks both add capacity. Which change do heads actually make?

Accepted Answer

Parallel heads inside one sublayer. A new block is another step in series.

Question 30

When can different heads in the same layer exchange information?

Accepted Answer

After their outputs are concatenated, through the output projection, then again in later blocks.

Question 31

What does a causal attention mask prevent?

Accepted Answer

A position attending to a later position.

Question 32

Which query-key pairs does a causal mask leave unblocked?

Accepted Answer

Pairs where the key is at the query position or earlier.

Question 33

How is a causal mask inserted into the attention calculation?

Accepted Answer

Blocked scores are set to a large negative value before the softmax.

Question 34

Is the causal mask a trained parameter?

Accepted Answer

No. It is a fixed pattern of which position pairs are allowed.

Question 35

A later real token and an earlier pad are both in the sequence. Which one does a causal mask block?

Accepted Answer

The later real token. An earlier pad stays visible until a padding mask blocks it.

Question 36

Why are pad positions added to a batch of sequences?

Accepted Answer

So shorter sequences can share a rectangular tensor with longer ones.

Question 37

What should a padding mask do when a pad position is used as a key?

Accepted Answer

Block that score so real queries do not mix in the pad's value.

Question 38

Does the padding id have a row in the token embedding table?

Accepted Answer

Yes. That id has a learned row, and the mask is what blocks it as a key.

Question 39

A decoder batch holds sequences of different lengths. Which masks does its self-attention need?

Accepted Answer

A causal mask for future positions and a padding mask for filler positions.

Question 40

How are pad positions usually treated by the next-token loss?

Accepted Answer

They are left out, so the model is not trained to predict the pad id as content.

Question 41

There is no positional encoding and no position bias. You reorder the input tokens. What does self-attention do?

Accepted Answer

It reorders its outputs the same way. Order is invisible to the layer.

Question 42

How does an absolute positional encoding enter the model?

Accepted Answer

A vector chosen by the index is added to the token embedding.

Question 43

What stops a learned absolute position table from serving an arbitrarily long sequence?

Accepted Answer

It has one learned row per index up to a fixed maximum, and no row past that.

Question 44

How is a sinusoidal absolute position vector built?

Accepted Answer

Fixed sines and cosines of the index fill the vector. Nothing in it is trained.

Question 45

Where does a relative position bias enter the calculation?

Accepted Answer

A term that depends on the offset between the query index and the key index is added to the score.

Question 46

The same token id sits at two indexes under absolute position vectors. What do those inputs share?

Accepted Answer

They share the token-embedding row and receive different position vectors.

Question 47

What does a linear distance bias add inside attention?

Accepted Answer

A per-head penalty on the score that gets more negative as the key moves farther away.

Question 48

How does rotary position encoding put order into the attention score?

Accepted Answer

It rotates the query and the key by angles that depend on their indexes, so the dot product depends on the offset.

Question 49

What does a residual attention sublayer return?

Accepted Answer

Its input plus the attention output.

Question 50

How does the residual skip help a gradient reach early layers?

Accepted Answer

The skip adds a copy of the gradient whose derivative is one, so that copy misses the factors inside the branch.

Question 51

The branch of a residual block outputs the zero vector. What does the block return?

Accepted Answer

Its input, unchanged.

Question 52

In a standard transformer block, which sublayers are added back onto the stream?

Accepted Answer

Both the attention sublayer and the feed-forward sublayer.

Question 53

Why must the attention branch match the residual stream in width?

Accepted Answer

The two tensors are added, and addition needs the same shape.

Question 54

Layer norm in a transformer computes its statistics over which axis?

Accepted Answer

The feature channels of one token. Each token is normalized on its own.

Question 55

Why is layer norm a better default than batch norm for token sequences?

Accepted Answer

Sequence lengths vary, so layer norm uses only the current token's channels.

Question 56

In the original block, what does layer norm receive?

Accepted Answer

The sum of the sublayer input and the sublayer output.

Question 57

When the norm sits on the branch input, what is added to the skip path?

Accepted Answer

The branch output, added to the original input that skipped that norm.

Question 58

After layer norm has made a token zero-mean and unit-variance, which learned parameters still apply?

Accepted Answer

A per-channel scale and a per-channel shift.

Question 59

Why does a pre-norm stack often apply one more layer norm just before the output head?

Accepted Answer

The residual adds were outside the branch norms, so the stream is scaled once more before the logits.

Question 60

How does the feed-forward sublayer treat two different positions?

Accepted Answer

The same map runs at each position, and those positions do not read one another.

Question 61

What is the shape of the original feed-forward sublayer?

Accepted Answer

A linear map up to four times the model width, a ReLU, then a linear map back to the model width.

Question 62

Inside one block, which sublayer mixes positions, and which mixes channels?

Accepted Answer

Attention mixes positions. The feed-forward sublayer mixes channels inside one position.

Question 63

In a classic block with feed-forward width four times the model width, where do more weights sit?

Accepted Answer

In the feed-forward maps, which outweigh the attention projections about two to one, ignoring biases.

Question 64

Is the feed-forward sublayer a convolution over neighboring tokens?

Accepted Answer

No. It has no window over neighbors. Each position is mapped on its own.