Context engineering basics

What is the context window?

The maximum number of tokens a model can process in one call, including both the input and the generated output.

What must fit in the token budget for one call?

The system prompt, examples, retrieved passages, conversation history, tool definitions and results, and the room reserved for the reply.

Does a model remember earlier calls?

No. Each call sees only what is in that call's input. Chat history works because the application resends it.

Why do token counts differ between models?

Each model has its own tokenizer, so the same text can split into a different number of tokens.

What is a system prompt?

Instructions set by the application that frame the model's role, rules, and format for the whole conversation.

What is few-shot prompting?

Including a few worked examples of the task in the prompt, so the model can follow the pattern.

What is zero-shot prompting?

Asking the model to do a task from instructions alone, without examples.

Why wrap inserted text in delimiters or tags?

To make clear where instructions end and data begins. The text is still in the prompt; tags only label it.

Where should key instructions go in a long prompt?

Usually near the start in the system prompt, and often restated near the end, since models use the start and end of long inputs most reliably.

What is the 'lost in the middle' effect?

Models tend to use information at the start and end of a long input better than information buried in the middle.

What is retrieval-augmented generation (RAG)?

Fetching relevant passages for each request and inserting them into the prompt, so the model can answer from them.

What is chunking?

Splitting documents into smaller passages for indexing and retrieval.

What is the chunk size tradeoff?

Small chunks are precise but can lose surrounding context. Large chunks keep context but dilute the match and use more tokens.

Why do neighboring chunks often overlap?

So a sentence or idea near a boundary appears whole in at least one chunk.

What is a text embedding?

A vector representing a piece of text, where texts with similar meaning end up close together.

Why must queries and documents use the same embedding model?

Different models produce vectors in different spaces, often with different sizes, so comparing them is meaningless.

What is cosine similarity?

A measure of how closely two vectors point in the same direction, from -1 to 1, used to compare embeddings.

What is a vector index?

A data structure that finds the nearest stored vectors to a query vector quickly, often approximately.

What is approximate nearest neighbor search?

Search that trades a little accuracy for large speed gains, such as HNSW graphs or IVF partitions.

What does top-k mean in retrieval?

The number of nearest results returned for a query.

Can vector search find a document added after the last index build?

No. A document must be embedded and indexed before search can return it.

What is BM25?

A keyword-based ranking function that scores documents by term matches, weighted by how rare the terms are and adjusted for document length.

What is hybrid search?

Combining keyword search and embedding search, so exact terms like codes and meaning-based matches both count.

What is reciprocal rank fusion?

A simple way to merge ranked lists by giving each item a score based on its rank in each list and summing them.

What is a reranker?

A model that reads the query and each candidate passage together and rescores them more accurately than first-stage search.

What does a high reranker score mean?

The passage looks relevant to this query by the reranker's judgment. It does not prove the passage is correct.

What is a metadata filter?

A rule restricting search to documents with certain attributes, such as product, date, or access level.

Why filter during search instead of after?

Filtering after taking the top k can leave few or no results. Filtering during search keeps k results that all pass.

Where should document permissions be enforced in RAG?

In retrieval, before any text reaches the model. Instructions in the prompt are not a security boundary.

What is query rewriting?

Rephrasing or expanding the user's question before searching, to better match how documents are written.

What is multi-query retrieval?

Searching with several versions of the question and merging the results, which widens coverage.

What does recall at k measure for retrieval?

The share of queries where a needed passage appears in the top k results.

What does precision at k measure?

The share of the top k results that are relevant.

Can adding more passages hurt an answer?

Yes. Irrelevant passages can distract the model or bury the useful one.

What goes wrong when a chunk says 'this limit' and the noun is in the previous chunk?

The model receives the number without knowing what it applies to. Adding headings or context to each chunk helps.

Why ask the model to cite sources?

So each claim can be traced to a passage the model was given, and unsupported claims are easier to spot.

What does it mean for an answer to be grounded?

Its claims are supported by the provided sources rather than by guesses.

How is conversation history kept within limits?

By dropping old turns, summarizing them, or storing key facts separately and reinserting them when needed.

What happens to a constraint that only appeared in a dropped turn?

The model no longer has it, unless it was summarized or saved and reinserted.

How does putting examples in a prompt differ from fine-tuning on them?

Prompt examples affect only calls that include them. Fine-tuning changes the weights, so it affects every call.

What is the difference between knowledge in the weights and in the context?

Weight knowledge comes from training and may be outdated or vague. Context knowledge is exactly the text supplied on this call.

What is prompt caching?

Reusing the computed state for a repeated prompt prefix across calls, which cuts cost and latency. It is not memory of past conversations.

What does the KV cache hold?

Keys and values for tokens already processed in the current generation, so new tokens do not recompute them.

What is a stop sequence?

A string that ends generation when the model produces it. It does not change the input.

What does the max output tokens setting do?

Caps how many tokens the model may generate. Too low cuts answers off; it also reserves space in the window.

Can a model reliably filter rows from a table it has only had described?

No. It only has the rows included in its input. Large data should be queried by a tool and the results inserted.

Why trim or summarize large tool results?

Raw outputs can fill the window with irrelevant detail, raising cost and making the useful part harder to find.

What is the tradeoff between long context and retrieval?

Long context is simpler and avoids missed retrieval, but costs more per call and can still miss details. Retrieval is cheaper but can fail to fetch the right passage.

How do you know a context change helped?

Score the system on a fixed evaluation set before and after the change, looking at answer quality, cost, and latency.