Transformer basics
An AI research scientist interview on the objectives, components, and training ideas a transformer paper treats as known.
These prompts are original practice. They are not questions from a named employer or a live hiring loop.
1.What does causal language modeling train the model to predict?
2.What loss is applied to the token that causal language modeling predicts?
3.How is perplexity obtained from the next-token cross-entropy?
4.Masked language modeling asks the model to predict which tokens?
5.What does span corruption hide?
6.Where may a prefix language model attend in both directions?
7.Which architecture is the usual home of masked prediction?
8.Which architecture is the usual home of next-token prediction?
9.How does an encoder-decoder transformer attend over source and target?
10.What does few-shot prompting change inside the model?
11.What does fine-tuning do with the weights?
12.What does LoRA train?
13.Does LoRA replace the tokenizer?
14.Is LoRA a change to the attention mask?
15.What does a full fine-tune update?
16.What is in-context learning?
17.What does teacher forcing use as the previous token during training?
18.What is exposure bias?
19.When is dropout active?
20.What does weight decay penalize?
21.Is weight decay the same object as dropout?
22.What two statistics does Adam keep from the gradient?
23.What does learning-rate warmup do over the first steps?
24.After warmup, what does a cosine learning-rate schedule do?
25.What does gradient clipping limit?
26.When is an ablation informative?
27.What symmetry does self-attention have if position signals are removed?
28.What bias does a convolution have about which positions can mix?
29.What does a recurrent net carry from one time step to the next?
30.How does a residual layer combine its update with the incoming vector?
31.Which product decides the attention pattern?
32.If the attention pattern is held fixed, what decides the vector written back?
33.An induction pattern sees A B earlier in the context, then sees A again. Which token does it attend to?
34.What is a probe?
35.What does a successful probe prove about a feature?
36.Does next-token pretraining, by itself, update the model when a prompt is later shown?
37.After softmax, what is one attention row for a single query?
38.In multi-head attention, how are key and value heads assigned?
39.In multi-query attention, how many key and value heads do the query heads share?
40.How does grouped-query attention share key and value heads?
41.Where is an absolute position signal added?
42.Where does a relative position bias enter the attention computation?
43.What does a rotary position encoding do to the query and the key?
44.What does a mixture-of-experts layer replace in a transformer block?
45.Does the mixture-of-experts router replace self-attention?
46.What does a load-balancing term push the router to avoid?
47.What does RMSNorm divide a vector by?
48.What does LayerNorm do that RMSNorm does not?
49.Where does pre-norm place the normalization?
50.Where does post-norm place the normalization?
51.What is the original transformer feed-forward?
52.What do SwiGLU and GeGLU replace?
53.Is evaluation perplexity under teacher forcing the same quantity as the quality of a sampled continuation?
54.What does a preference method such as RLHF do with comparisons?
55.What does a KL term do in a preference update?
56.What does temperature change when a model is decoding?
57.Which vocabulary items can top-k sampling still draw?
58.Which vocabulary items can nucleus sampling still draw?
59.What do a byte-level vocabulary and a subword vocabulary change?
60.What is fertility?
61.Can bidirectional attention use a token to the right of the query?
62.Can causal attention use a token to the right of the query?
63.How is the label aligned in next-token training?
64.Why is predicting the current token the wrong alignment for next-token training?
65.Which of these is a set of three different regularizers?
66.What does label smoothing do to the target?
67.What does a larger depth add?
68.What does a larger width increase?
69.What scale is applied to query-key scores before the attention softmax?
70.Before the output projection, how is the attention result formed from the values?
71.Do all queries in one head share a single attention distribution?
72.In cross-attention, where do queries, keys, and values come from?
73.In self-attention, where do queries, keys, and values come from?
74.Inside a standard block, where do different positions mix?
75.Why does the feed-forward put a nonlinearity between its two linear maps?
76.What does the output softmax of a language model distribute over?
77.What are logits, before the vocabulary softmax?
78.For a one-hot next-token target, what is the cross-entropy?
79.What does greedy decoding pick at each step?
80.What does beam search keep while decoding?
81.What does a key-value cache store during decoding?
82.Why can teacher-forced next-token training score every position in one forward pass?
83.What is the difference between a padding mask and a causal mask?
84.What dimension does a standard residual block return?
85.How are queries, keys, and values obtained from the residual stream?
86.What does the attention output projection do with the heads?
87.How is multi-head attention different from adding depth?
88.Which axes does LayerNorm use for its mean and scale?
89.How does byte-pair encoding build a subword vocabulary?
90.Under a subword vocabulary, what can happen to one word?
91.What can a byte-level vocabulary represent without an unknown-token bucket?
92.If fertility rises, what happens to a fixed context window?
93.Does a typical transformer weight update revise the tokenizer merge table?
94.What are padding, beginning-of-sequence, and end-of-sequence markers?
95.What is an end-of-sequence token used for during generation?
96.How does the number of query-key scores in full self-attention grow with sequence length?
97.How does the cost of a position-wise feed-forward grow with sequence length?
98.Where is the masked-language-modeling loss computed?
99.During a parallel teacher-forced pass, does a causal model leave the future visible?
100.What does raising the temperature do to the vocabulary distribution?
101.Do temperature, top-k, and nucleus sampling change attention weights?
102.After a token is drawn at decoding time, what happens to it on the next step?
103.Does a zero-shot prompt, with no examples, update the weights?
104.Does a forward pass by itself change the weights?
105.How does Adam combine its two moments in the update direction?
106.How does AdamW apply weight decay relative to the adaptive step?
107.What does a relative position bias depend on?
108.After rotary positions are applied, what does the query-key score depend on through that rotation?