Transformers

1.Step 40,000 on train-a100-07 dies with CUDA out of memory. This morning the max length went from 2,048 to 8,192. The parameter count is the same as the run that fit yesterday. Where did the extra GPU memory go?

2.A note on the length change says memory will double because 8,192 is twice 4,096, so the same box can take the new max if one other job is deleted. This run materializes the score matrix. What does the note get wrong?

3.Length goes from 2,048 to 8,192, and the batch is cut from 16 to 4 so the token count per step stays put. The score matrix is still materialized. What happens to score memory?

4.Score memory has to stay about flat while length goes from 2,048 to 8,192. The kernel materializes the scores. Today's batch is 16. Which batch holds that term flat?

5.The cluster takes train-a100-07 back at 18:00 if the job has not started. Someone wants to skip the one-step memory profile and launch all 40,000 steps at length 8,192. That shape has never run. What do you run first?

6.The 8,192 run runs out of memory. The 2,048 run of the same width fit with room to spare. A change request cuts the hidden size so the parameter shard gets smaller. What is wrong with that as the first move?

7.Yesterday's 8,192 step fit. Today's image was rebuilt without the fused attention kernel, and the same step runs out of memory in the score allocation. Weights, batch, and length are unchanged. What came back?

8.On the 8,192 run the forward pass finishes and the backward pass runs out of memory. The peak is the tensors retained for the backward. Parameters loaded cleanly. What are you looking at?

9.The trainer config says max length 8,192. The step log's length histogram never goes past 2,048, and a second cap in the collator is still 2,048. The long-prompt canary of this checkpoint falls. What did the run actually train?

10.config.max_position_embeddings is 2,048. The loader now feeds 8,192, and this checkpoint's training recipe has no position scaling. The forward does not throw. Why should the run stop?

11.The memory sheet for the 8,192 job adds batch, length, hidden, and layers, and says the card has room. Eager attention is on, and the step runs out of memory. What did the sheet leave out?

12.train-a100-07 runs out of memory in the backward at step 40,000. On-call starts writing the weights to disk on every step, hoping the card will clear. The live tensors do not shrink. What would drop the backward peak?

13.Activation checkpointing is on for the 8,192 job. The step fits, and the step time is about a third higher because the forward runs again during the backward. Monday's freeze needs this job to finish. What do you put in the plan?

14.The process runs out of memory while loading Adam onto the GPU, before the first forward. Activation checkpointing is turned on and the same load is rerun. What do you expect?

15.Every transformer block is wrapped in activation checkpointing, and the step still runs out of memory while building one block's score matrix at length 8,192. The fused kernel is off. Why can't a finer boundary fix that allocation?

16.Checkpointing every block makes the 8,192 job miss Monday. The extra recompute pushes the finish past the freeze. A profile shows a segment of four blocks still fits on the card. What do you change?

17.The long-context job runs out of memory at step 40,000. The yaml it loaded is the debug file, and that file sets activation checkpointing to false. The production yaml sets it to true. Length is 8,192. What ran?

18.serve-l4-2 runs out of memory once several long chats are live. The change under test turns on activation checkpointing in the serve process. Weights and the cache dtype stay the same. What does that flag do here?

19.Attention is wrapped in activation checkpointing. The feed-forward is still saved for the backward. At length 8,192 the profile peak is the feed-forward activation, and attention is no longer the largest tensor. What is still stored?

20.Checkpointing is on, the fused kernel is on, and the job still runs out of memory. The largest tensors in the profile are the Adam moments, one full copy on every rank. What is the next move?

21.One image turned on activation checkpointing and fp16 together. The loss is NaN by step 200. The previous image, checkpointing off and fp32, was finite. What do you run next?

22.The 8,192 job does not fit in fp32 activations. The card supports bf16. Matmuls and activations move to bf16, and a short trial runs before the 40,000-step launch. What is that trial for?

23.The job was finite in fp32. The new image runs the matmuls in fp16 with no loss scaler, and the loss is NaN by step 200. The data has not changed. What is missing?

24.The fp16 scaler logs an overflow at step 40,012 and is about to skip the update and cut the scale. A patch applies those gradients anyway so the step counter moves. What can those gradients write?

25.The fp16 run dies at step 40,000. The resume file has the weights and Adam and does not have the loss-scale value. The first resumed steps overflow and skip. What should have been in that file?

26.After step 40,000 the fp16 loss is flat, the step counter climbs, and the grad norm in the fp16 copy is zero. The scaler's scale has fallen to a tiny value. What is the run doing?

27.Autocast runs the matmuls in fp16, and the custom loss then casts the logits to fp16 before the cross-entropy. Long rows go NaN even with a scaler on the backward. Where does the reduction stay?

28.The 8,192 job is over the card. TF32 is enabled in the hope that activation memory drops by half, the way a 16-bit cast would. The peak does not move. Why?

29.The H100 job trains in bf16. A checklist from the old V100 fp16 run blocks launch until a GradScaler config is copied over. A short bf16 trial is already finite. What do you do with that block?

30.The mixed-precision trial is finite and fits. Launch review wants the step-200 loss to match the old fp32 run bit for bit before anyone spends 40,000 steps. What gate do you put in front of that review?

31.This run keeps an fp32 master and uses bf16 for the matmuls. The file written at step 40,000 is the bf16 cast only. A resume from that file jumps the loss. What was left out?

32.The run dies at step 40,000. The newest file has the weights and a step field of 40000, and it has no Adam moments. The loader will open it. What can you honestly use it for?

33.A loader puts step 40,000 onto a fresh Adam whose moments are zero, then continues at the late learning rate. The first updates come out around the size of the learning rate and ignore how small the gradient is. Why?

34.Weights and Adam load, and the logged step is 40,000. The scheduler in the new process starts at zero, so the learning rate is the warmup peak rather than the late cosine value. The moments are the old ones. What disagrees?

35.The job is killed while writing the step-40,000 checkpoint. The file is short and will not load. Step 38,000 in the same bucket is complete. What do you resume, and what do you change in the writer?

36.The job died 12 steps after the step-40,000 save, so those 12 steps will be redone. A proposal saves the full Adam state on every step so a redo cannot happen. That save already stalls the GPUs. What do you keep?

37.The step-40,000 checkpoint is a ZeRO shard set written by 8 ranks. The only free allocation this morning is 4 GPUs of the same type. The 4-rank job is pointed at the 8 shard files. What fails?

38.All 8 ranks write their state to the same checkpoint path at step 40,000. The file will not load, and the bytes look interleaved. Step 38,000 was written by rank 0 alone and loads. What do you change?

39.Weights, Adam, and step 40,000 restore. The sampler starts at epoch 0, offset 0. The learning rate is the late cosine value. What will the next thousands of steps train on?

40.The pool reclaims train-a100-07 and wipes its local SSD. The only step-40,000 checkpoint lived on that disk. Durable storage has step 30,000. Besides resuming 30,000, what changes on the next attempt?

41.Production serves the exponential moving average of the weights. The crash file at step 40,000 holds the raw weights, and the average was not saved. A canary of that file moves on the long-prompt slice. What did the canary load?

42.The run that reached step 40,000 clipped the grad norm at 1.0. The resume config turns clipping off. The first long batch after resume spikes the loss. The moments loaded. What setting was part of the run you meant to continue?

43.The scheduler preempts the job just after the step-40,000 save. That file has weights, moments, schedule, scaler, and sampler. The replacement machines are the same 8 GPUs. A script in the image starts the new allocation at step 0. What should it do?

44.The step-40,000 save dies because the disk is full. The optimizer file is most of the bytes. A quick fix writes weights only, which fits, and deletes the moment files to free the volume. What can you no longer do?

45.A packed row is 2,048 tokens and holds ticket 18441, then ticket 18442, with an end-of-sequence id between them. Attention is one causal mask over the row. The draft of 18442 quotes a dollar amount that exists only in 18441. What did the packer allow?

46.The packer inserts an end-of-sequence id between documents and leaves a single causal mask over the packed row. A review says that special id is the boundary, so no mask work remains. What does a later token still do?

47.The pack mask is block-diagonal, so ticket B cannot see ticket A. Position ids still run from 0 through 2,047 across the row. Ticket B's first token sits at position 1,300. At serve, that same text starts at position 0. What is mismatched?

48.The packed run's loss sits well below the padded baseline after the same number of steps. Nothing else was supposed to change. Attention across the document boundary has not been measured. What do you check before calling the model better?

49.Before packing, a step held 8 sequences and about 4,000 real tokens. After packing it still holds 8 sequences and about 16,000 real tokens. The learning rate was tuned for the old step, and the new loss is lower. What changed besides the padding?

50.The serve process packs two live customer prompts into one sequence so it can reuse the training packer, with one causal mask. Both customers are in that row. What can the second customer's next token see?

51.The packer stores each document's length in a side tensor. The attention call never receives it and runs one causal mask over the concatenated row. A test reads the first token of the second document. What do you expect on the first document's positions?

52.The attention mask stops at the document boundary. The label shift does not. The last token of ticket A is trained to predict the first token of ticket B. Serve will never ask A to continue into B. What do you change in the loss?

53.Ticket 19002 is 3,000 tokens. The pack target is 2,048. The packer cuts it after 1,800 tokens and fills the rest with the start of ticket 19003. Serve truncates a too-long ticket to 2,048 and does not append another ticket. What is that train row?

54.Each document's mask is a block of ones. The packer concatenates those blocks along the sequence and uses the result as a dense mask. A probe at the first token of document 2 shows attention on document 1. Why?

55.A batch of 16 has fifteen notes under 80 tokens and one note of 2,048. Over half the allocated tokens are pad. A request is open for a second GPU because the step is slow. What do you change before you add the GPU?

56.Every training batch is padded to the model max of 8,192. This split's 99th percentile length is about 700. Memory tracks the 8,192 rectangle all day. What should the allocation follow?

57.Pad positions use the ignore index, and the loss number looks right. The memory profile still shows a full padded rectangle. A note says the ignore index returned those bytes. What did it return?

58.The loss ignores pads. The attention mask is ones everywhere, so real tokens attend to pad positions. Short notes in the canary depend on how the pad id was embedded. What is still open?

59.Training padded on the right, and position ids followed the real tokens. A new collator left-pads so the batch looks like the server. The ids are still a plain range over the padded width, so a short note now sits at the high positions. What moved?

60.Length buckets are 128, 512, and 2,048. A 513-token note falls in the 2,048 bucket and is padded by about 1,500 tokens. Most notes in that bucket are just over 512. Where is that waste from?

61.Pad positions are a real label, the pad id, because the ignore index was never set. Short rows in the canary end by emitting that id. Long rows, which have little padding, look fine. What did the short rows learn?

62.The epoch is sorted by length so padding stays low, and nothing shuffles inside a length band. Early steps are almost all short notes. Late steps are almost all long notes, and the loss rises into the freeze. What did the sort make?

63.Train loss ignores pad positions. The accuracy number used for promotion counts them, and a pad position is marked wrong. Short rows look far worse than the loss. What is that accuracy counting?

64.Documents are packed up to 2,048 real tokens, so padding should be rare. The collator then pads every packed row to the model max of 8,192. The memory profile matches the old padded run. Where did the pack go?

65.Training pinned tokenizer revision 1.4 and stored its hash on the checkpoint. serve-l4-2 loads the bucket file named latest, which is revision 1.6 with a different hash. The weights loaded. What should the process do?

66.Canary answers are fluent, the lookup does not throw, and the content is wrong. The server tokenizer hash is not the training hash. A proposal raises the temperature to loosen the bad answers. What do you compare first?

67.A new special token was inserted at id 3, and every later id shifted by one. The serve image ships last week's embedding, trained before the insert. Requests do not crash. What do the shifted ids read?

68.A tool token was appended to the tokenizer. The embedding was not resized, so its row count is the old vocab. The first request that emits the new id returns a 500 from the lookup. When should that failure have happened?

69.The training pipeline prepends the beginning-of-sequence id. The server template also prepends it. A golden prompt that should start 12, 48 arrives as 12, 12, 48. The long-prompt slice moves. Where should that id be added?

70.The tokenizer hash matches the checkpoint. The server template no longer wraps the role tokens the training job put around every example. Word ids match on a bare sentence. The canary of real prompts moves. What did the hash not cover?

71.Batched generate on serve-l4-2 marks a row finished when any position holds the end-of-sequence id. The pad id is that same id, and short rows are left-padded. Those short rows finish immediately. A batch of one, with no pad, works. What is colliding?

72.Revision 1.6 changes the unicode normalizer. The ASCII canary is flat. Notes with accented place names score worse, and encode() on those names disagrees with revision 1.4. What is too narrow?

73.A faster tokenizer build cuts encode time on serve-l4-2. On the regression corpus it disagrees with the pinned tokenizer on whitespace before punctuation. The latency chart improved. What accepts the build?

74.encode() matches the pinned tokenizer. Users still see replacement characters in names. The server decode path has byte fallback off, and the training tokenizer has it on. Round-trip of those names fails only on the server. What else has to match?

75.The checkpoint embedding has 32,008 rows. The tokenizer can emit id 32,004. config.vocab_size is 32,000, and the server builds the module from the config, then drops the extra rows on load. The first added id crashes. What should boot refuse?

76.The promotion harness tokenizes the eval set with whatever tokenizer is installed on the laptop. The candidate artifact is revision 1.4. The laptop has 1.6. The score is what the review would ship on. What did that score use?

77.serve-l4-2 is fine at startup. After lunch, with 12 live chats of a few thousand tokens each, the process dies allocating cache. The weights have not changed since the boot that fit. What grew with the chats?

78.The 7B bf16 weights are about 14 GiB on the 24 GiB L4. At this model's layer and head layout, one live sequence of 8,192 tokens is about 1 GiB of KV cache. The release allows 16 live sequences at that length. What fails the budget?

79.Paged attention is on, so a short chat no longer reserves a full 8,192-token block. Afternoon traffic is all long transcripts, and the live tokens themselves still exceed the card. A note says paging will compress those tokens. What does paging reduce?

80.One release wants a new checkpoint and an fp8 KV cache that halves cache bytes. In past trials, cache precision moved the long-prompt slice on its own. The long-prompt slice is also the slice for the new checkpoint. How is this release cut?

81.The checkpoint uses grouped-query attention with 8 key-value heads. The server cache config was copied from an older model and allocates one key head per query head, 32 of them. The replica runs out of memory at a concurrency the sheet allowed. What was sized wrong?

82.Chats have already returned, and GPU memory does not fall. The next wave runs out of memory even though the in-flight count looks low. Cache bytes stay high while those request ids are finished in the log. What is still held?

83.The replica caches KV for the system prompt and keys that cache only by model id. Tuesday's release changes the system prompt text. New chats still follow the old instruction. The weights did not change. What is the hit returning?

84.Prefix cache is allowed for the shared system prompt. A bug caches the whole request, user document included, under the model id and the length. The next request with that length hits. What can the hit contain?

85.The replica allows 8 live requests. Eight chats of about 200 tokens fit in a daytime test. The morning batch is eight transcripts of about 16,000 tokens, and the cache allocation fails. What did the limit of 8 count?

86.The budget counted prompt tokens, and prefill of a full batch fits. Generation may run for 1,024 new tokens. The process runs out of memory around reply token 900. What was left out of the budget?

87.Request A finishes and its sequence slot is given to request B. B's logical length is set back to zero, but the kernel still reads key rows out to A's old length. B's reply quotes A's document. What stayed live?

88.The offline scorer pads each batch to the longest prompt in it and allocates a rectangular cache. One 8,192-token transcript beside short prompts blows memory for the whole batch. What keeps one long row from setting everyone else's width?

89.In a static batch, short replies are returned only when the longest generation in the batch finishes. Their tail latency is that long reply. The SLO is per request. What changes the tail?

90.An 8,192-token prefill runs in the same iteration as in-flight decodes. Decode steps that should be short wait on that prefill, and both the new long prompt and the in-flight replies miss their timeouts. What keeps decode moving?

91.Two requests are waiting, one 200 tokens and one 4,000. A helper concatenates them into one sequence with an end-of-sequence id between them and calls that a batch. Under one causal mask, what can the second request attend to?

92.The serve config sets max batch size to 32 and max length to 8,192. On this L4, 32 sequences at that length do not fit the KV budget, and 32 sequences of a few hundred tokens do. What limit matches the card?

93.The nightly scorer sorts by length and batches similar rows, which cuts its padding. That waiter is copied onto the user-facing path. Short requests sit until a similar-length partner arrives, and they miss the SLO. Where does the waiter belong?

94.One row in a mixed batch emits the end-of-sequence id. The loop marks the whole batch finished and frees every row's cache. The other replies stop mid-sentence. What should end?

95.Mixed-length prefill is left-padded. The mask is a plain causal triangle over the padded width, with no pad mask. Short rows attend into the pad positions on their left. Batch size one, which has no pad, scores fine. What is missing?

96.Mixed-length prefill is right-padded. The server takes the last index of each row as the state decoding starts from. On a short row that index is a pad. Batch size one works. What should it read?

97.Admission was sized from the daytime average prompt, about 512 tokens, and a batch that fits at that average was allowed. Morning traffic is a burst of 8,192-token packets. The cache fills and the replica runs out of memory. What did the average hide?

98.Checkpoint 40000 is canaried on 5% of traffic. The mixed-set score matches production. The long-prompt slice, the one this release said it would hold, is down 7 points. The band is 2 points. Short prompts are up 1. What happens to the canary?

99.Checkpoint 38000 is up for promotion. The long-prompt slice moved 0.4 and the short slice moved 1.1, both inside the 2-point band. The tokenizer hash matches training. The KV budget fits on the production L4 at the batch you will serve. What is the call?

100.The long-prompt score moved against production. The canary decoded at temperature 0.7. The production number was measured at temperature 0. The template and the tokenizer match. What do you know?

101.Long prompts in the canary hit the gateway timeout. The scorer stores a timeout as zero. Short prompts match production. The weights may be fine, and the long slice is still red. What happens before any promotion talk?

102.The canary artifact pairs the new weights with a tokenizer hash that is not the training pin. The long-prompt slice is the score that moved. Short prompts moved less. What is this canary worth?

103.Long-prompt quality holds on a borrowed 80 GiB card. Production is the 24 GiB L4. At the batch and length in the release notes, the KV budget does not fit beside the weights on that L4. The architecture is unchanged. Does this promote onto the L4?

104.Step 40,000 has the lowest training loss of the run. On the frozen long-prompt set, step 30,000 scores better and sits inside the band against production. Step 40,000 is outside that band. Which file can move?

105.The offline long-prompt slice has already missed the 2-point band. A plan puts that checkpoint on 50% of traffic to see whether users notice. The tokenizer matches the pin. What does the offline miss do?

106.The canary is at 5%. The long-prompt online slice crosses the rollback line that was set before the canary started. The mixed average is still inside its band. What do you do this hour?

107.Last week's baseline called a prompt long above 1,000 tokens. This week's candidate uses 4,000 tokens for the same label. The prompts in the slice are not the same set, and the score moved. What has to be frozen before the two numbers mean anything?

108.One release changes the checkpoint, the served max length, the tokenizer revision, and the live batch. The long-prompt canary moves, and there is no way to tell which lever did it. What is allowed to move toward production?