Context engineering basics
An MLOps interview on the token budget, the prompt cache, retrieval indexes, and the KV cache that serves a context.
These prompts are original practice. They are not questions from a named employer or a live hiring loop.
1.What counts the token budget of a served request?
2.Two fields have the same character length. One is a sentence and the other is a long identifier. What can the serving tokenizer report?
3.A team plans capacity with one token per four characters. What is that ratio when the server enforces the window?
4.A context window is set to 8192. What does 8192 limit?
5.A request includes a system prompt, history, retrieved chunks, and a reserved answer length. How many token budgets do those parts use?
6.A million chunks sit in the index, and this request copies none of them into the prompt. What do those chunks cost this window?
7.Two requests run at the same time, and their prompts do not share a prefix. Do they draw from one token budget?
8.The server reserves 600 tokens for the answer before it has generated any. When does that reservation take room away from the prompt?
9.The window holds 4096 tokens and the server reserves 512 for the answer. How long can the prompt be?
10.The server reserved 400 tokens for the answer, and the reply used 90. What are the 310 that were never generated?
11.System text and history already use 3000 tokens, and 500 are reserved for the answer, inside an 8192 window. How many tokens are left for retrieved chunks?
12.Each chunk is wrapped with a heading and a source line before it is inserted. What do those lines cost?
13.The chat template inserts role markers around the user text. Do those markers count toward the budget?
14.Two retrieved chunks overlap, and both copies are placed in the prompt. How does the shared text count?
15.The rendered prompt is past the window. What does truncation do to the extra tokens?
16.Truncation might drop the oldest history, the middle, or the retrieved text. Why does the server have to define which?
17.The truncation policy drops the oldest history. Which turns remain?
18.The truncation policy drops the middle of the conversation. What is kept?
19.The truncation policy drops retrieved text first. What happens to the conversation?
20.Retrieval returns a chunk, then truncation removes that chunk before the model runs. What does the model attend over?
21.Why should the server truncate with the serving tokenizer's ids rather than with a character slice?
22.The same system prompt is sent on every request. What does that do to each request's window?
23.Which tokenizer may encode a prompt for this set of weights?
24.The same sentence is passed through two versions of the serving tokenizer. What can change?
25.The server has the language-model weights and is missing the tokenizer files. What can it not reproduce?
26.A tokenizer vocabulary contains 128000 ids. What does 128000 measure?
27.The weights and the vocabulary stay fixed, and the chat template changes. What changes in the served request?
28.Why can the word rain with a leading space be a different token from rain with no leading space?
29.A common word is one token. A rare product code of similar length is eight tokens. Why?
30.A byte-level tokenizer meets a character that is not one vocabulary word. What happens to that character?
31.What does the language model attend over when a prompt is served?
32.A token count was saved last week. The tokenizer artifact has since been replaced. What should the server do with that saved count?
33.The answer reservation is written as 256. What is that 256?
34.Truncation cuts on a token boundary. What can a reader see in the remaining text?
35.Server config names a special token that the tokenizer vocabulary does not contain. What is that name, at serve time?
36.A tool definition is written out at the front of the prompt. What does that text cost?
37.At what point is the token budget the real budget?
38.A chunk is retrieved for a request. Where does it live, and how does the model see it?
39.How is a retrieval index different from the KV cache?
40.What is the embedding model version, relative to a vector index?
41.The language-model tokenizer is replaced. The embedding model that built the index is not. Which queries still match the index?
42.An index was built with embedding model v3. The server embeds the new query with v4. What is wrong?
43.Which quantities set the bulk of the vector storage in a flat index?
44.The corpus gains a large batch of new documents, and they are indexed. What grows?
45.The same chunk text is re-embedded with a larger embedding dimension. What changes for the index, and what does the prompt still pay?
46.A metadata filter excludes a chunk. When does that exclusion take effect?
47.A chunk from the wrong project is already in the prompt, and a filter runs after that. What is true?
48.Search is asked for the top 4 chunks. What does that 4 limit?
49.A vector search hits a chunk. What does the prompt receive from that hit?
50.The vector index stops answering. The model server is healthy. What kind of failure is that?
51.Retrieval times out, and the server is configured to continue. What prompt is served?
52.Retrieval times out, and the server is configured to refuse the request. What reaches the model?
53.The index returns successfully with zero hits. How is that different from a timeout?
54.A source page is edited, and the index is not updated. What can a later query return?
55.The embedding model is replaced with a new version. What has to happen before queries use the new model against this corpus?
56.Half the vectors in one index came from embedding model A and half from model B. What is wrong with neighbor search?
57.A chunk is copied into the prompt. What should the trace record about that chunk?
58.One index holds several tenants, and the query omits the tenant filter. What can be copied into this prompt?
59.A reranker keeps 3 of 20 retrieved chunks. What does the language model see?
60.The index stores product-quantized vectors. What did that change?
61.A document is deleted in the source store. The index is not updated. What can search still do?
62.The server embeds the user question in order to search. Where does that query vector go?
63.A retrieved chunk's vector is small, and the chunk text is 3000 tokens. The remaining window has room for 1000 tokens. What decides whether the chunk fits?
64.What does prompt caching reuse?
65.The system prompt is shared, and a user-specific block is appended after it. Which part can be the shared prefix?
66.When can two requests share a prefix cache?
67.A request hits a cached copy of the system-prompt prefix. What still has to run?
68.A user id is inserted as the first tokens, ahead of a system prompt that would otherwise be shared. What happens to prefix sharing across users?
69.The first tokens of a cached prefix are edited. What happens to the cached suffix of that prefix?
70.Two prompts share their first 400 tokens and differ at token 401. What can a prefix cache reuse?
71.The same two chunks are inserted in the opposite order. Why can that miss a prefix cache of the old prompt?
72.Two users have different account ids, and those ids never appear in the prompt. Both prompts start with the same system-prompt tokens. What can they share?
73.Where do source documents live for retrieval?
74.A request misses the prompt cache. What does that miss mean?
75.A cached prefix is evicted. The next request sends the same tokens. What work comes back?
76.Two different prefixes are both kept in the prompt cache. How is that state stored?
77.Why does the system-prompt or skill version belong in the trace next to the model version?
78.A prompt-cache hit reuses a prefix of 2000 tokens. What happens to the context window?
79.Turn 1 is already a cached prefix. This request adds a new user message as turn 2. What is turn 2, relative to that cache?
80.Temperature is changed and the prompt tokens stay the same. What can still be reused?
81.Two model versions are asked to serve the same token prefix. Can they share one prefix cache?
82.Two prompts differ in their first tokens and then share a long span in the middle. What can a prefix cache reuse from that span?
83.What does the KV cache hold for a request that is being served?
84.A KV-cache byte count has a leading factor of 2. What is that 2?
85.Model A has 24 layers and model B has 48. Prompt length and KV-head shape match. Which served KV cache is larger?
86.A serving config keeps the same prompt and the same head dimension, and it reduces the number of KV heads. What happens to KV-cache bytes?
87.The head dimension doubles. Layers, KV heads, sequence length, and element width stay the same. What happens to KV-cache bytes?
88.How does KV-cache memory grow as the served sequence gets longer?
89.A request finishes, and its KV cache was not kept as a shared prefix. What can happen to that memory?
90.How many bytes does one bf16 element of the KV cache use?
91.How many bytes does one fp32 element of the KV cache use?
92.What does prefill do with the prompt?
93.In ordinary single-token decode, what does one step do with the KV cache?
94.A prompt grows from 1000 tokens to 3000 on the same weights. What happens to prefill?
95.A longer prompt is served on unchanged weights. What grows, and what stays the same?
96.Several sequences of different lengths are in one server. What does paged attention change about KV-cache allocation?
97.KV-cache quantization is turned on. What changes?
98.KV-cache quantization is enabled. Which artifacts keep the precision they already had?
99.The loaded position method supports 8192 positions. What is the longest max-context value this server can honestly use?
100.An operator raises max context in the server config from 8192 to 32768 and changes nothing else. What did that change?
101.Prompts in one batch have different lengths. What do padding and packing each do?
102.Several requests are packed into one sequence. What must the attention mask do?
103.A long history is replaced, in the next prompt, by a shorter summary of that history. What is the summary?
104.A summary is appended, and the original turns are still in the sequence whose KV cache the server is holding. What happens to that cache's memory?
105.The server logs the full prompt of a request. What text is inside that log?
106.Those full prompt logs contain retrieved text and user text. What kind of choice is the retention period?
107.What is speculative decoding, relative to the context this server serves?
108.What is flash attention, on a server that already has a KV cache and a retrieval index?