Context engineering basics

Why count tokens on the server with the model's own tokenizer?

Estimates from characters or another tokenizer can be off, causing requests to fail or be truncated unexpectedly.

Why pin the tokenizer version with the model?

A different tokenizer gives different counts and IDs, breaking budgets, caches, and outputs.

What should a server do when a request exceeds the context limit?

Either reject it with a clear error or apply a documented truncation policy. It must never silently cut arbitrary text.

Why must the server define which part truncation drops?

Dropping the oldest history, the middle, or retrieved text gives very different answers. Callers need to know which.

Why return a flag when input was truncated?

So callers and traces know the model did not see the full input.

What does the model attend over when a prompt is served?

Only the tokens actually sent in that request, after any truncation. Nothing else on the server is visible.

What does a prompt cache store?

The computed KV state of a prompt prefix, so later requests with the same prefix can skip that prefill.

What is the key for a prompt cache entry?

The exact token sequence of the prefix, along with the model version. One changed token early on means a miss.

Two different prefixes are both in the prompt cache. How is that stored?

As separate entries, though a tree structure can share the tokens they have in common at the start.

What is prefix sharing with a radix tree?

Storing cached prefixes in a tree so requests that share a beginning reuse the same KV blocks.

What is a prompt cache TTL?

How long an unused entry stays before it is evicted. Short TTLs save memory but lower the hit rate.

What eviction policy do prompt caches often use?

Least recently used: when memory runs short, drop the entries that have gone unused the longest.

Why monitor the prompt cache hit rate?

Hits cut latency and cost. A sudden drop often means a prompt change put variable text before the stable prefix.

Why is putting the current time at the top of a system prompt costly?

It changes every request, so the prefix never matches and the cache never hits.

What happens to cached prefixes when the model weights change?

They are invalid, since KV values depend on the weights. The cache must be cleared or keyed by model version.

How many bytes does one fp32 element of the KV cache use?

Four. bf16 and fp16 use two, and fp8 uses one.

How much KV cache does one token need?

2 × layers × KV heads × head dim × bytes per element, for the key and value in every layer.

What sets how many concurrent long requests a GPU can serve?

The memory left after the weights, divided by the KV cache each request needs at its length.

What is KV cache offloading?

Moving inactive KV blocks to CPU memory or disk and loading them back when needed, trading latency for capacity.

When is a request's KV cache freed?

When the request finishes, unless the server keeps its prefix in a prompt cache for reuse.

What does building a vector index involve?

Chunking documents, embedding each chunk, and inserting the vectors and metadata into an index structure.

Why pin the embedding model version?

Query and document vectors must come from the same model. Upgrading it means re-embedding the whole corpus.

How do you switch embedding models safely?

Build a new index with the new model alongside the old, evaluate it, then switch traffic with an alias.

What is an index alias?

A stable name that points to one index version, so swapping or rolling back is a pointer change.

What is an incremental index update?

Adding, changing, or deleting only the affected chunks, instead of rebuilding the whole index.

A document is deleted at the source but not in the index. What can search still do?

Return its old chunks, which can expose removed or outdated content.

What is a tombstone in an index?

A marker that a record was deleted, so searches skip it until the space is reclaimed by compaction.

Why give chunks stable IDs?

So updates and deletes can find the right records, and citations and logs point to the same chunk over time.

What is index freshness lag?

The time between a source change and when search reflects it. It should be measured and have a target.

When does a metadata filter take effect?

At query time, when the search runs. Pre-filtering restricts candidates before the vector search, post-filtering removes results after.

Why can post-filtering return too few results?

If most of the top k fail the filter, few or none remain.

How are document permissions enforced in retrieval?

Store access labels with each chunk and filter on the caller's identity in every query.

What do ANN search parameters such as ef or nprobe trade off?

Recall against latency: searching more of the index finds more true neighbors but takes longer.

Why shard a vector index?

To spread a large index across machines for memory and speed. Queries go to all shards and results are merged.

Why replicate a vector index?

For more query throughput and to survive a node failure.

Why batch embedding calls when indexing?

Batches use the GPU or API far more efficiently, cutting cost and time for large backfills.

Why cache embeddings by content hash?

Unchanged chunks do not need re-embedding on each rebuild, which saves cost.

What retrieval metrics should be monitored in production?

Query latency, empty-result rate, filter hit rates, index size, freshness lag, and error rate.

What can a rise in the empty-retrieval rate mean?

A broken filter, an index swap that lost data, or new queries that the corpus does not cover.

Why keep a labeled query set for the retriever?

To measure recall@k on each index or model change before it goes live.

What is in a full prompt log?

Everything sent to the model, including user text and retrieved documents, which may contain personal or restricted data.

Full prompt logs contain retrieved and user text. What kind of choice is the retention period?

A policy choice about privacy, compliance, and debugging needs, not just a storage setting.

What can be logged instead of full prompts?

Token counts, chunk IDs, prompt template version, and hashes, which support debugging without storing raw text.

Why log the prompt template version with each request?

So output changes can be traced to the template that produced them.

Why set a server-side cap on output tokens?

To bound latency, cost, and KV memory for each request.

What does streaming change for the client?

Tokens arrive as they are generated, lowering perceived latency. Total generation time stays the same.

What is admission control for long requests?

Rejecting or queueing requests when KV memory is short, instead of letting them fail midway.

Why separate queues for short and long prompts?

Long prefills can delay many short requests. Separate queues or chunked prefill keep short latency low.

What usually dominates the cost of a RAG request?

Input tokens from retrieved passages and history, which are paid on every request unless cached.