Context engineering basics

Which unit decides whether more text fits in the window?

Tokens, as counted by that model's tokenizer, not characters or words.

Why must the budget leave room for the output?

Input and generated tokens share the same window. A full prompt leaves no space for the answer.

One paragraph from a handbook on disk is inserted. How much of the rest of the handbook did the model see?

None of it. The model sees only what is in the window.

Does the model remember the previous request?

No. Each call is independent. Continuity comes from resending history in the new context.

What makes instructions easier for a model to follow?

Being specific, stating the output format, giving the reason for constraints, and keeping them free of contradictions.

Why say what to do rather than only what not to do?

A clear target behavior is easier to follow than a list of bans, which can leave the right action unstated.

Why mark inserted documents with tags or delimiters?

So the model can tell instructions apart from data and know where each source begins and ends.

What do in-context examples teach the model?

The expected format, style, and decision boundary. They work best when they cover varied, realistic cases.

How can examples bias the output?

If most examples share one label or length, the model leans toward that label or length.

What does assigning a role in the system prompt do?

Sets tone, expertise level, and focus, such as 'you are a careful claims reviewer', which shapes later answers.

Why specify the output format precisely?

So outputs can be parsed and compared, and so the model does not add or drop fields.

What is prefilling the response?

Starting the assistant's reply with fixed text, such as an opening brace, to steer format.

Why give the model room to reason before answering?

Writing intermediate steps often improves accuracy on multi-step tasks, at the cost of more output tokens.

Why retrieve passages instead of inserting the whole corpus?

The corpus is larger than the window, and irrelevant text costs tokens and can distract the model.

Where can a length-based splitter place a boundary?

Anywhere the length limit falls, including mid-sentence. It does not look at meaning.

What is a structure-aware splitter?

One that cuts at headings, paragraphs, or sentences, so chunks keep their meaning intact.

What are contextual chunk headers?

Adding the document title or section name to each chunk, so the chunk still makes sense when read alone.

What does an embedding model produce for a chunk?

A fixed-length vector placed so that texts with similar meaning have nearby vectors.

Why embed queries and chunks with the same model?

Vectors from different models live in different spaces, so their similarities are meaningless.

What is the trade-off in choosing k for retrieval?

Larger k raises the chance the right passage is included, but adds tokens, cost, and distracting text.

What does a reranker add?

A more careful relevance score for each retrieved candidate, used to reorder them and keep only the best.

Where does pure vector search often fail?

Exact identifiers, codes, rare names, and numbers, where keyword search does better.

A source document was updated but not re-indexed. What does search return?

The old chunks. The index reflects the last build, not the current source.

Where should access control apply in retrieval?

In the retriever, before passages reach the model, based on the user's permissions.

How is retrieval measured separately from the answer?

With labeled query and relevant-passage pairs, using recall@k or similar, before any generation.

How is the answer measured separately from retrieval?

Give the model the correct passages and check whether its answer is correct and supported by them.

What is a grounded answer?

One whose claims are all supported by the provided context.

Why ask for citations to passage IDs?

Reviewers and code can check each claim against its source, and unsupported claims become visible.

Why tell the model what to do when the context lacks the answer?

Otherwise it may guess from its training. An explicit 'say it is not in the sources' instruction reduces made-up answers.

How is long conversation history kept within the window?

Keep recent turns in full, summarize or drop older ones, and keep standing facts in a short persistent note.

A constraint appeared only in a turn that was dropped. What happens?

The model no longer sees it and can violate it. Important constraints belong in the system prompt or a kept note.

Why trim large tool results before adding them?

Raw outputs can be huge. Keeping only the needed fields saves tokens and keeps the model focused.

What does the KV cache hold?

The attention keys and values for tokens already processed in the current request, so they are not recomputed for each new token.

During a long completion the KV cache grows. What is being added?

Keys and values for each newly generated token, in every layer.

How is a prompt cache different from a KV cache?

A prompt cache keeps the processed state of a common prefix across separate requests. The KV cache lives within one request.

What must match for a prompt cache hit?

The exact same token prefix from the start. Any change early in the prompt makes everything after it a miss.

A new system line is appended after content already processed. What happens to the cached part?

The earlier processed tokens can be reused, since the prefix is unchanged. Only the appended tokens need processing.

How should a prompt be ordered to benefit from caching?

Stable parts first, such as system prompt, tools, and fixed examples, with changing parts such as the user request last.

Why place key instructions at the start or end of a long prompt?

Models tend to use the beginning and end of long contexts better than the middle.

Can adding more context make answers worse?

Yes. Irrelevant or conflicting passages can distract the model, and longer prompts raise cost and latency.

Two retrieved passages disagree. What should the context make clear?

Each passage's source and date, so the model can say which is current or report the conflict.

Why not ask a model to compute over a large table pasted as text?

Models are unreliable at exact counting and filtering over many rows. Running a query with a tool is safer.

How does putting examples in a prompt differ from fine-tuning on them?

Prompt examples affect only calls that include them and cost tokens each time. Fine-tuning changes the weights once.

When is putting a whole document in context better than retrieval?

When it fits comfortably and the task needs reading across the whole document, such as summarizing or comparing sections.

Why version prompts like code?

Small wording changes can shift behavior. Versions let you compare results and roll back.

How do you know a context change helped?

Run the old and new versions on the same fixed evaluation set and compare the scores, not a few hand-picked examples.

What is overfitting a prompt?

Tuning wording to pass a small set of test cases so closely that it fails on new inputs.

What is a stop sequence?

A string that ends generation when the model produces it, used to cut output at a known boundary.

What happens when the max output token limit is hit?

Generation stops mid-output, which can leave structured output such as JSON incomplete.