Transformers

1.Job H-14 trains grouped-query attention on the deduped 2024 crawl and multi-head attention on the raw 2022 crawl. Validation loss is 2.41 versus 2.58. The abstract assigns the 0.17 gap to grouped-query attention. What did the two configs fail to share?

2.The depth memo locks 12 layers against 24, each on 40 billion tokens. The 24-layer job was later extended to 110 billion tokens; the 12-layer job had already stopped. Final loss favors the deep run. What did the extension add on only one side?

3.Figure 2 marks validation loss at 120 million, 1.1 billion, and 6.4 billion parameters and draws a straight line on log-log axes. The caption says that line is the loss at 40 billion parameters, a size with no run. What is the line anchored to?

4.The mixture sheet lists pylon-mcq.jsonl as 1.2 percent of pretraining tokens. The headline number is 71 percent accuracy on that same file. The talk calls the items unseen. Which tokens did training already include?

5.Keel-R trains rotary positions at length 8,192. Keel-A trains ALiBi at length 2,048. On a 1,024-token prose set, Keel-R has the lower perplexity. The slide says the positional scheme was the only difference. Which training knob moved with the scheme?

6.The 3 March protocol names the baseline as this lab's reimplemented 6-layer decoder, trained on this corpus. Table 1 instead scores a public 12-layer checkpoint that another group fine-tuned. The new block wins that comparison. Which bar did 3 March set?

7.On the sealed clause set, exact match fell from 41 to 36 while token F1 rose from 62 to 67. That evening, after both columns were filled, the analysis note names token F1 as the primary endpoint. What had been locked before the columns existed?

8.The variant row shows seed 17 only, with perplexity 0.4 below the control. The methods paragraph says the drop was stable across initializations. Which other initializations did seed 17 get compared with?

9.A linear probe on the layer-11 feed-forward state predicts subject-verb number at 84 percent on the Nave sentences. The code never reads an attention weight. The section title says the attention heads encode agreement. What did the 84 percent attach to?

10.Splice-6 deletes the layer-9 attention block and, on the same launch line, sets residual dropout from 0 to 0.1. Held-out loss rises from 2.11 to 2.27. The caption says the deleted block had been carrying the loss. How many factors did that launch move?

11.The local-attention run discards every training document longer than 1,024 tokens. The full-attention control keeps those documents. Both are scored on short news leads, where local attention wins. The note credits the window. What else did the local run change?

12.The layer-6 feed-forward is removed, and the same config flips every remaining feed-forward from SwiGLU to GELU. Holdout perplexity goes from 14.2 to 16.0. The figure label reads 'cost of the layer-6 feed-forward.' What second edit is inside that cost?

13.Cobble plots loss at 10 to the 18, 10 to the 19, and 10 to the 20 training FLOPs and calls the picture a measured scaling exponent. No fourth budget was trained, and the fit was not checked on a held-out size. What does that exponent summarize?

14.Kiln is a code-completion set. A search finds 64 of its 80 reference solutions as verbatim files in the GitHub scrape used for pretraining. The Kiln pass rate is reported as generalization to new code. What did that search find in the scrape?

15.Absolute learned positions are trained at length 512, the width of that position table. Rotary positions are trained at length 4,096. Perplexity at length 512 favors the learned table. The abstract says learned positions win at matched length. What was actually matched?

16.The registered plan says the control is the lab's own dense 1 billion model, November weights, same tokens. The plot's gray bar is a sparse mixture-of-experts model released by another group. The new attention block beats the gray bar. Which control did the plan name?

17.Bits per byte on the full prose holdout got worse under the new block. Accuracy on a 40-sentence slice, chosen after that worse number was known, got better. The camera-ready reports only the 40-sentence accuracy. When was the slice designated?

18.The seed plan lists five initializations. The paper's table prints seed 3. A footnote says the other four are pending, and the abstract, published while they were still pending, says the gain replicated. What has been repeated?

19.On the Silt tapes, copy accuracy falls when the layer-8 feed-forward outputs are zeroed. Attention outputs are left untouched. The caption calls layer 8 an induction head. What did the zeroing change?

20.Plinth-D is 32 layers at width 768. Plinth-W is 12 layers at width 1,408. Parameter counts match near 250 million, and Plinth-D has the lower loss. The talk says depth won with width held fixed. What was not held fixed?

21.The linear-attention run uses a 50,000-piece vocabulary. The softmax control uses the 32,000-piece vocabulary named in the plan. Loss favors linear attention, and the sentence under the table credits the kernel. Which other config field differs?

22.Removing the attention residual at layer 3 is paired, in that arm only, with a peak learning rate cut from 3e-4 to 1e-4. That arm's loss is higher. The reviewer reply says the residual was necessary. What else did the reply's arm change?

23.At a fixed 300 million parameters, loss is plotted after 2 billion, 8 billion, and 32 billion tokens, and a power is fit. The draft calls the power the data-scaling law of this architecture. No budget past 32 billion was trained. What does the fit summarize?

24.Orchard grammar items in the test file were emitted by make_orchard.py, the same script, seed, and templates that built the synthetic pretraining split. Test accuracy is 93 percent. The paper calls Orchard a held-out grammar. What did the training split share with the test file?

25.One arm changes the ALiBi slopes and packs documents to 8,192 tokens. The control keeps the old slopes and packs to 2,048. Loss favors the new slopes. The caption says only the slopes changed. What did the packer change?

26.Methods say the positional baseline is the sinusoidal run defined in section 2, trained this quarter. The gray bar in Figure 4 is last year's production checkpoint, after an extra fine-tune on edited news. The new scheme beats the gray bar. Which run was section 2's baseline?

27.Five probes were scored: agreement, tense, coreference, negation, and animacy. Only animacy beat its control. After the sheet was sorted, animacy was marked primary, and the paper reports a successful probe. Which probe was free to become the reported one?

28.Seed 42 is the only variant seed in Figure 1. The lab book shows seeds 43 and 44 finished with worse loss than the control and were left out of the figure. The caption says the variant improved loss. Which finished seeds is that sentence using?

29.A histogram of absolute values in the layer-10 feed-forward activations is peaked near zero. No attention map is computed. The slide says attention in this model is sparse. What is the histogram a picture of?

30.The 24-layer run warms up for 10,000 steps. The 12-layer control warms up for 2,000 steps. Token budget and width match, and loss is lower for the deeper run. The abstract credits depth. What schedule difference sits beside the layer count?

31.Grouped-query training packs documents into full sequences. The multi-head control pads each document to the same length and masks the pad. Loss favors grouped-query, and the memo credits the query grouping. What data-layout choice moved with the grouping?

32.Arm D deletes layer 4 and unties the input and output embeddings. The control keeps layer 4 and keeps the embeddings tied. Arm D has worse perplexity. The sentence says deleting layer 4 hurt. What second change is inside arm D?

33.Three trained points, at 1e18, 1e19, and 1e20 FLOPs, are connected and labeled the compute-optimal frontier. No other model size was trained at those budgets, and no other budget was trained at those sizes. What would a frontier of that name have required?

34.Quay is a word-problem set. The pretraining forum dump contains 91 of the 100 Quay problems with the worked solutions still attached. Quay accuracy is reported as reasoning on new problems. What was attached to those problems in the dump?

35.The no-position run is trained at length 256. A note says positions are unnecessary, so the context can stay short. The rotary run is trained at length 2,048. Losses match on a 128-token eval. The slide says the no-position scheme matches rotary positions. What did the note change besides the scheme?

36.The plan says the baseline is a 12-layer model trained here for 50 billion tokens of this corpus. Table 3's baseline cell is a perplexity copied from a 2019 paper's published table. The new run beats that copied number. What does the copied cell fail to show?

37.Perplexity on the sealed prose set rose from 13.1 to 13.6. That afternoon the authors adopt a public split they describe as easier, and they report a rank on that split. The sealed set is not the rank. When was the public split adopted as the reported result?

38.The new block's one seed beats the control by 0.02 nats. A pilot on an earlier block, printed in the appendix, saw seed-to-seed swings of about 0.05 nats. This block was not repeated. The abstract calls the 0.02 gap a reliable gain. What does the pilot's swing say about that gap?

39.Patching the full layer-7 residual stream from a clean run changes the number prediction. The patch replaces attention output, feed-forward output, and the skip together. The caption says head 7 suppresses the wrong number. What did the patch fail to separate?

40.The 36-layer run uses batch size 2,048. The 12-layer run uses batch size 512. Both see 40 billion tokens, width matches, and loss favors the deep run. The talk credits the layers. Which training-scale choice differed?

41.The gated-attention run's data card is 40 percent English. The standard-attention card is 70 percent English. The other listed fields match. A multilingual perplexity average favors gated attention, and the title credits the gate. What did the data card change along with the gate?

42.The arm without the first-layer attention block uses a constant learning rate. The control, which keeps the block, uses cosine decay. Loss is worse without the block. The caption credits the block's presence. Which schedule sat only on the ablated arm?

43.Loss falls from depth 6 to depth 12 to depth 24, close to a straight line, at matched width and matched tokens. The closing sentence says returns to depth stay linear at depth 48 and at depth 96. Neither of those depths was trained. What did the line measure?

44.Marrow's test items are long passages with a question under each one. The pretraining corpus includes those passages with the questions and the answer strings still in the text. Marrow accuracy is called comprehension of unseen documents. In what form did those passages appear in pretraining?

45.Rotary base 10,000 is trained at length 2,048. Rotary base 500,000 is trained at length 8,192. Both are scored at length 2,048, and the large base wins. The note says the base was isolated. What did the large-base run train on that the other run did not?

46.The plan says the baseline is softmax attention at the same width, with no extra path. The plotted baseline adds a depthwise convolution beside attention, left in from a debugging config. The new variant beats that hybrid. Which model did the plan say to beat?

47.On the Cinder bitext, BLEU fell for all four language pairs. chrF rose for one pair and fell for the other three. After the table was printed, the results paragraph says the study's metric is chrF on the pair that rose. What did that paragraph select?

48.Seeds 2 and 5 diverged at step 20,000. They were restarted at a lower learning rate, and those restarts are absent from the paper. Seed 1, which did not diverge, is the only reported variant seed. The conclusion says the variant is robust to initialization. What happened to the seeds that diverged?

49.Saliency here is the gradient of the correct logit with respect to the input embeddings, computed on Wicket coreference items. No attention matrix is stored. The figure caption says the model attends to the antecedent. What was differentiated?

50.The 24-layer model uses rotary positions. The 12-layer model uses learned absolute positions. Width, token count, and sequence length match, and loss favors the deep model. The heading is a single word, Depth. What scheme difference sits under that heading?

51.The control is scored on split V0, drawn in January. The attention-variant run is scored on V1, a new draw made the week that run finished. Loss on V1 is lower. The note says the variant beat the control on the validation set. How many validation draws are in that comparison?

52.Block 7 is removed only in arm Q. Arm Q also sets weight decay to 0.1, while the control uses 0.01. Arm Q's loss is higher. The slide says block 7 was doing useful work. Which hyperparameter moved with the deletion?

53.Downstream accuracy at 1, 4, and 16 billion tokens is 61 percent, 72 percent, and 74 percent. A power is fit through those three accuracies, and the paper quotes the exponent. The gains are already shrinking. What is the fit being asked to describe?

54.Cinder's test file holds 2,000 sentence pairs. A join finds 1,400 of them, source and target together, inside the training bitext. The paper still calls Cinder an unseen translation test. What did the join match?

55.Relative bucket biases are trained at length 128. Rotary positions are trained at length 2,048. At a 64-token eval the bucket model is close to the rotary model. The caption says the buckets are sufficient, so training length does not matter. What comparison is that caption using?

56.The plan's baseline is an untuned model after the same number of pretraining steps, with no Quay answers in its training. The reported baseline was fine-tuned on Quay answers before the comparison. The new model beats that fine-tune on Quay. What did the plan say the baseline must not train on?

57.Accuracy on the full Vellum memo set was 54 percent for both models. Next-token loss fell on the shortest quartile of memos and rose on the longest. The next day's note defines success as that short-quartile loss. What had the full-set accuracy done before the note was written?

58.Methods say the paper reports the median across seeds. The artifact store contains one variant checkpoint, seed 7, and no other variant seed. Table 2 prints seed 7's loss and labels it the median. What is that median taken over?

59.On the Brine strings, copy accuracy falls when every feed-forward output is zeroed at inference. Attention outputs are left as trained. The paper says attention implements the copy. Which path did the zeroing leave intact?

60.The deep model has 16 layers and 8 heads per layer. The shallow model has 8 layers and 16 heads per layer, so the product of layers and heads matches. Loss favors the deep model. The abstract says depth was isolated from the attention layout. What still differs inside each attention block?

61.Softmax attention is trained with residual dropout at 0. The linear-attention arm uses residual dropout 0.1 together with the linear kernel. That arm has higher loss, and the caption blames the kernel. Which dropout setting belongs only to the linear arm?

62.The ablation script deletes layer 5 and sets dropout on every remaining block to 0.2. The control keeps layer 5 and uses dropout 0. Loss rises. A bullet calls the run a layer-5 ablation. How many edits does the script apply on the ablated arm?

63.Three widths are plotted against loss. The small model trained for 8 epochs, the middle model for 3, and the large model for 1, all on the same finite corpus. The caption calls the curve parameter scaling. What else changes as the parameter count grows?

64.A training shard named eval_mirror is a character-for-character copy of the sealed prose test, included by a bad copy command. Test perplexity is very low. The draft says the model generalized to the sealed test. What is eval_mirror relative to that test?

65.Both positional schemes are trained at length 512. At evaluation, the rotary run is scored with a 2,048-token window and the ALiBi run is scored at 512. Rotary perplexity is lower. The slide says training length was controlled, so the scheme won. Which length was left uncontrolled?

66.The plan's baseline is run B0: 12 layers, tied embeddings. The figure labels the word baseline on run B3: 13 layers, untied embeddings. The new model beats B3. Which run did the plan assign that word to?

67.The preregistration has a blank line labeled primary metric. Four metrics were computed. One p-value fell under 0.05. That metric's name was typed into the blank the next morning, and the paper says the study was preregistered. What was on the blank when the metrics were computed?

68.The baseline's three seeds land at losses 2.10, 2.18, and 2.21. The variant has one seed at 2.14. The paper says the variant beats the baseline. Where does 2.14 sit among the baseline seeds?

69.The paper says attention at layer 2 was measured. The job log shows probe_ffn.py reading the second feed-forward's hidden state. The attention tensor is never written to disk. What did that script read?

70.The 24-layer run trains with label smoothing 0.1. The 8-layer run trains with label smoothing 0. Token counts match. The smoothed training loss looks lower for the deep run, and the caption drops the smoothing detail and credits depth. Which training target differed?

71.The attention-variant run sorts training documents from short to long before the first step. The control shuffles those same documents. Final loss favors the variant. The paragraph credits the attention change and never mentions order. What order did only the variant train on?

72.The layer-2 attention ablation also freezes the token embeddings on that arm. The control keeps training the embeddings. Loss is worse on the ablated arm. The ticket title is 'layer-2 attention removed.' What else did that arm stop updating?

73.A scaling figure has three dots. The small dot and the large dot are dense transformers. The middle dot is a sparse mixture-of-experts model, plotted at its active-parameter count. One curve through all three is called the dense scaling law. What is the middle dot?

74.Twelve-grams from the sealed test appear about 400 times in the training text. The authors note that the training documents have different titles, and they conclude the test cannot be in the training data. What did the title check leave untouched?

75.Scheme S is trained at 2,048 tokens and tested at 8,192. Scheme L is trained at 8,192 and tested at 8,192. Scheme L scores higher on the long test. The caption credits the scheme and not the training length. What training length did Scheme S never see?

76.The registered baseline is full attention. The config key window was left at a debugging value of 256, and that run is the one the table labels baseline. The full-attention variant beats it. What context did the labeled baseline actually train with?

77.Full-set exact match for the new block is lower than the control. The reported metric averages only items the new block already got right in a smoke test. That item list was written down after both full-set scores were known. On the list, the new block looks perfect. Which items were left out of the average?

78.The preregistered seed list is 11, 12, 13, 14, and 15. The hypothesis said loss would fall. Seeds 12 through 15 finished higher than the control. Seed 11 fell, and it is the only seed in the paper. What did the paper do with the four seeds that rose?

79.The number logit changes when the whole layer-5 block is replaced by the block from a counterfactual run. Heads inside the block are not swapped one at a time. The text says the change shows which head carries number. What swap is missing from that claim?

80.The 36-layer model received a learning-rate sweep and then trained at the best rate from that sweep. The 12-layer model trained at the original rate and received no sweep. Loss favors the deep model. The paper credits depth. Which model received the sweep?

81.Grouped-query attention is trained with a feed-forward expansion of 8. The multi-head control uses an expansion of 4. Loss favors grouped-query. The conclusion says the key-value head count produced the gain. Which width differed between the runs?

82.At decoding time on the sealed clause set, the layer-9 block is zeroed and the sampling temperature is raised from 0 to 0.8, both only on the ablated transcript. Exact match falls. The note calls the fall the effect of removing the block. How many decoding changes does that transcript contain?

83.The scaling line uses three sizes. The largest was halted at half its planned tokens after the loss stalled. The two smaller runs finished their budgets. The line still passes through the halted point and is called the fitted law. What budget did the largest point miss?

84.The coding eval asks for functions from the private lumen-ledger repository. That repository is one of the pretraining code dumps, and several reference functions appear byte for byte in the dump. The writeup calls the eval a test of unseen private code. What did pretraining already contain?

85.The extrapolated-position run trains at 16,384 tokens. The rotary control trains at 4,096. Both are scored at 4,096, and the long-trained run wins. The title says the extrapolation rule is what won. Which run already covered the score length in training?

86.The plan identifies the baseline by git commit c4e1, a model trained on the shared crawl. The table's baseline is a colleague's model trained on that colleague's private crawl. The new run beats the colleague's loss. Which crawl did commit c4e1 train on?

87.Micro F1 and accuracy both favored the control. That evening the label set was regrouped into coarser classes, and macro F1 on the new classes favored the variant. The paper reports that macro F1. When were the coarser classes defined?

88.The caption prints plus or minus 0.03, across seeds, beside the variant's one loss. The 0.03 figure is copied from a different experiment's seed table. This variant has a single seed in the store. What does the printed 0.03 describe?

89.For each token, the analysis records the absolute position of the largest layer-12 feed-forward activation and averages those positions. The average is larger in later layers. No attention weight is read. The paper says attention distance grows with depth. What position was recorded?

90.The 32-layer model uses a byte-level vocabulary of 65,536. The 12-layer model uses a wordpiece vocabulary of 32,000. Loss in nats per byte favors the deep model. The slide credits depth and says the per-byte unit makes the tokenizers comparable. What does that unit leave in place?

91.The plan says to remove one sublayer: the layer-12 attention block, leaving the layer-12 feed-forward in place. The launched arm removes both the attention block and the feed-forward at layer 12. Loss rises, and the arm is described as the planned one-block ablation. How many sublayers did the launch remove?

92.Three loss points are fit in log space. With all three, the slope is negative. Drop the largest run and the slope on the remaining pair is positive. The paper reports the three-point slope as the scaling law and does not mention the sign change. What happens to the sign without the largest run?

93.Three model sizes have seed-loss intervals 2.40 to 2.55, 2.42 to 2.60, and 2.45 to 2.58, from three seeds each. The intervals overlap. The caption ranks the sizes in a strict order. What do those intervals say about the ranking?

94.The four demonstration items in the few-shot prompt are copied from the pretraining set, answers included. The test items that follow the prompt are new. Accuracy is called few-shot learning from unseen demonstrations. What had training already shown the model?

95.The learned position table has 512 rows, so that model cannot train past length 512. Rotary positions train at 4,096. Perplexity at 512 favors the table. The reviewer response says this isolates the scheme at the only length the table supports. Which rotary training run is missing?

96.The plan's baseline is 32 layers at width 1,024. The first job to finish was 24 layers at width 1,280, and that job was entered as the baseline because it finished first. The variant beats it. Which shape did the plan call the baseline?

97.Eight language pairs were scored. The variant won three and lost five. Only the three wins appear in the results table, under a sentence calling them the target languages. The note that lists those target languages is dated after the scoring grid. What did the table drop?

98.Baseline seed losses are 2.05, 2.11, and 2.16. The variant has one seed at 2.12. The paper sets 2.12 beside 2.16 and writes that the variant beats the baseline. Which baseline seed is that sentence using?

99.Binding is scored as the cosine between the final-layer hidden state and a gold entity vector. The cosine rises under the new block. No attention pattern is computed. The abstract says the block improved attention-based binding. Which two vectors entered the cosine?

100.The 48-layer run clips gradients at 0.1. The 12-layer run clips gradients at 1.0. The rest of the written recipe matches, and the deep run's loss is lower. The caption credits the added layers. Which clip threshold did the deep run use?

101.The arm that deletes the layer-1 block draws the remaining weights from a normal distribution with scale 0.02. The control uses scale 0.05 and keeps the block. The deleted arm loses. The note says the block was the only difference. Which initialization scale did the deleted arm use?

102.Splice-2 removes the layer-10 block and switches that arm from pre-norm to post-norm. The control stays pre-norm and keeps the block. Loss is worse for Splice-2. The caption says the layer-10 block is what mattered. Which normalization order did Splice-2 introduce?

103.Mean attention entropy is lower in the variant than in the control. The probe log contains that entropy and no other field. There is no coreference label anywhere in the log. The title says the variant's attention tracks coreference chains. What did the probe record?

104.The attention statistic averages weights among positions 1 through 32 and ignores every later position. Training sequences are 2,048 tokens. The paper says the statistic shows long-range attention. Which span did the average actually use?

105.The probe averages attention across all 16 heads and emits one entropy. The writeup says head 6 is the coreference head. Head 6 is never read on its own. What did the average combine?

106.Attention maps for number agreement are computed on 500 English sentences. The conclusion says those maps are how the model marks agreement in the other languages in the training mix. No sentence outside English was probed. Which sentences were mapped?

107.One run uses a causal mask and the other a bidirectional mask. The only stored output is the logit on the final token. No attention matrix is written. The discussion says those logits show that attention looks only leftward. What was stored?

108.The error bar drawn on the variant is the standard deviation of minibatch losses inside seed 1. Seed 1 is the only seed that exists. The bar is captioned uncertainty across runs. What does the bar vary over?