AI/ML system design

What is the difference between functional and non-functional requirements?

Functional requirements say what the system does. Non-functional ones say how well: latency, availability, cost, privacy, and scale.

What is the request path?

The sequence of steps a request passes through, such as gateway, retrieval, model, post-processing, and response.

What does an API gateway do?

Receives requests and handles authentication, rate limits, routing, and sometimes caching before passing them on.

What is a latency budget?

The total time allowed for a request, split across each step so the sum meets the target.

What is time to first token?

How long until the first output token arrives. It sets how responsive a streaming answer feels.

Why stream model output?

Users see text as it is generated, which feels faster even though total generation time is the same.

What is the throughput-latency tradeoff?

Larger batches serve more requests per second but make each request wait longer, especially the slowest ones.

What is dynamic batching?

Grouping requests that arrive close together into one model call to use the hardware more fully.

What is continuous batching for LLMs?

Adding and removing sequences from a running batch at each generation step, instead of waiting for the whole batch to finish.

Besides the weights, what uses GPU memory during LLM serving?

The KV cache for every sequence in flight, which grows with batch size and sequence length.

What is paged KV cache management?

Storing the KV cache in small fixed-size blocks, like memory pages, so memory is not wasted on reserved but unused space.

What is the difference between batch and online inference?

Batch scores many inputs on a schedule and stores results. Online scores each request as it arrives.

When should predictions be precomputed?

When inputs change slowly and the set of things to score is known, so lookups are fast and cheap.

What is a feature store?

A system that computes, stores, and serves features consistently for training and serving, often with point-in-time lookups.

What are online and offline feature stores?

The offline store holds history for training. The online store serves the latest values with low latency for live requests.

What is training-serving skew?

A mismatch between how features or inputs are produced in training and in serving, which hurts live quality.

What is a two-stage retrieval and ranking design?

A fast stage retrieves a few hundred candidates from millions, then a more accurate, slower model ranks them.

What is candidate generation?

The first stage of a recommender or search system, which quickly narrows a huge catalog to a manageable set.

What are the main stages of a RAG pipeline?

Ingest and parse documents, chunk, embed, index, retrieve for each query, optionally rerank, build the prompt, generate, and check the answer.

What happens in document ingestion?

Fetching, parsing, cleaning, deduplicating, chunking, embedding, and indexing documents, usually as a background job.

What must happen when the embedding model changes?

Every document must be re-embedded with the new model and a new index built before queries use it.

Why deduplicate documents before embedding?

To avoid paying to embed copies and to stop duplicates from crowding search results.

Why add diversity rules to retrieval results?

So several near-identical chunks from one source do not crowd out other useful sources.

How often should an index be rebuilt?

As often as the content changes and freshness matters, using incremental updates when a full rebuild is too slow.

What is a model router?

A component that sends each request to the most suitable model, for example a small model for easy requests and a large one for hard ones.

What is a model cascade?

Trying a cheap model first and passing only uncertain or failed cases to a more expensive one.

What is a fallback?

A simpler response used when the main model fails or times out, such as a rule-based answer, cached result, or smaller model.

What is load shedding?

Rejecting or degrading some requests when the system is overloaded, so the rest still meet their targets.

What is rate limiting?

Capping how many requests or tokens a client can use in a time window, to protect shared capacity.

What is backpressure?

Signaling upstream callers to slow down when a downstream component cannot keep up.

Why put a queue in front of a model service?

To absorb bursts and decouple callers from the model's speed. Queues need limits, or waits grow without bound.

Why run long tasks as background jobs?

So a user request does not hold a connection open for minutes, and work can be retried and tracked separately.

What are common caches in LLM systems?

Exact response caches for repeated requests, embedding caches, retrieval caches, and prompt-prefix caches.

What belongs in a response cache key?

Everything that affects the answer: the normalized input, prompt template version, model version, and index version.

What is a semantic cache?

A cache that returns a stored answer for a new request that is similar in meaning, not just identical. It risks wrong hits.

What is autoscaling?

Adding or removing servers based on load signals such as queue length, latency, or GPU use.

What is a cold start for a model server?

The delay while a new server loads weights and warms up before it can serve traffic quickly.

What is a readiness check?

A check that a server is fully loaded and able to serve before the load balancer sends it traffic.

Why run several replicas of a model server?

To handle more traffic and to keep serving when one replica fails.

What is model parallelism?

Splitting one model across several devices, by layers or by slicing weight matrices, when it does not fit on one.

Why serve a quantized model?

Lower-precision weights use less memory and bandwidth, so more requests fit per device and responses are faster.

What should be retried when a later step fails after the answer was generated?

Only the failed step. Retrying the whole call regenerates the answer, costs more, and can give a different result.

How should personal data be handled in logs?

Masked or removed before storage, with access limits and retention rules.

What is tenant isolation?

Keeping each customer's data, indexes, and caches separate, so one tenant's content never reaches another's requests.

What are input and output guardrails?

Checks before and after the model, such as blocking unsafe requests, filtering personal data, and validating answers against sources or schemas.

What should be observable in an ML system?

Request rates, latency percentiles, errors, fallback use, token counts, cost, input and output distributions, and quality signals.

What should be versioned in a deployed ML system?

Models, prompts, feature definitions, indexes, and configuration, so any response can be traced and any change rolled back.

How can a model be switched without downtime?

Load the new version alongside the old one and move traffic over gradually or all at once, keeping the old one ready to switch back.

How do you estimate serving cost?

Cost per request, from tokens or compute time on the real hardware, times expected traffic including peaks.