LLM serving

Why pin the engine next to the weights?

vLLM, SGLang, and TensorRT-LLM change schedulers and kernel support between versions. The image digest and the weight revision are one release.

What is FastAPI's job in front of a model?

HTTP, auth, and request validation. It should call a server or a batcher. Putting generate() inside the web worker ties a GPU to one process and one request.

What does BentoML package?

The model, the Python environment, and a service definition that can be run as one unit. It is the packaging. The throughput still comes from the engine inside the service.

When is a service framework more than FastAPI?

When you need the model load, the adaptive batch, and the image build described in one place. FastAPI alone leaves those as scripts beside the route.

What does Ray Serve add?

Replicas of a deployment, a request router, and a way to put a Python preprocessing step in front of the GPU replica. The replica still has to call an efficient engine.

When is Ray Serve the right outer layer?

When one product has several models, CPU pre and post steps, and you already run Ray. A single LLM with one engine does not need a second scheduler in front of vLLM's.

What is NVIDIA Triton Inference Server?

A server that loads a model repository and exposes HTTP or gRPC, with dynamic batching. It is not the Triton language that compiles GPU kernels. The names collide.

What does tritonclient speak?

The inference protocol of Triton Inference Server: model name, version, and tensors. It does not load weights. A chat completion client will not speak this without an adapter.

What is a Triton model repository?

A directory per model with a config and the versioned files. The server polls or loads that directory. Shipping a new version is a new folder, not an in-place overwrite.

What do you operate when the engine is vLLM?

The image, the max batched tokens, the tensor-parallel size, the quantization flag, and the GPU count per replica. Those set capacity. The Python client only sends requests.

Why is the quantization flag part of the deploy?

An AWQ file and an FP8 file need different kernels. Pointing the server at the wrong flag fails the load or silently runs a slower path.

Where does a GGUF file belong in production?

On a local or CPU-offload runner such as llama.cpp. A GPU cluster serving many tenants should use the format that cluster's engine kernels implement.

What does prometheus-client export?

Counters and histograms your process updates, which a scraper pulls. The server must increment token counts and queue time. A scrape of process CPU is not that.

Which SLIs matter for an LLM replica?

Time to first token, inter-token time, tokens per second, fraction of requests rejected, and KV cache occupancy. Request count alone hides a full cache.

What does pynvml read?

NVIDIA driver counters: memory used, utilization, temperature, and power, per GPU. It explains a slow replica. It does not tell you which request filled the cache.

How do you split GPU memory on an LLM replica?

Weights, then the KV cache pool, then activations. If weights take the whole card, the server has no room to batch, and throughput collapses to one short sequence.

When does a GPU data loader such as DALI belong in the path?

When image or video decode on the CPU cannot feed the GPU. Tokenized text for an LLM is usually small enough that the loader is not the bottleneck.

When is a GPU array library the right preprocess?

When a NumPy transform on the CPU is the long pole and the op maps to an array kernel. It does not replace the inference engine.

What is a cold start for an LLM replica?

The weights loading onto the GPU, plus kernel or graph warmup, before the first real token. A scale-up that is slower than the traffic spike misses the spike.

Why keep a minimum replica count?

So one failure or one scale-down does not leave zero loaded copies. Loading tens of gigabytes is not a health check you want on the request path.

How do you drain an LLM replica?

Stop new requests, finish or shed the ones in the KV cache, then exit. Killing the process drops every sequence that was mid-decode.

How do you roll a new engine image?

One replica at a time, with the same weights, watching time to first token and error rate. A new image can change kernels and break a quantization flag.

How do you roll a new weight revision?

Load it beside the current revision if you can, or drain a replica onto the new files, then move traffic. The tokenizer and template move with the weights.

How do you ship a new LoRA without reloading the base?

Only if the engine supports hot adapter load and the adapter matches the resident base revision. A mismatched base is a wrong model, not a small update.

Where do private weights live?

In a store the node pulls at start, with a short-lived credential. A weight file baked into a public image is a leaked model.

What do you check before a weight file is pulled in prod?

The license and the use restriction on that revision. A Hugging Face id is not permission to serve it to customers.

What of a prompt do you keep?

A request id, the model revision, token counts, and latency. The raw prompt is customer data. Retention and access for it are a separate decision.

What does a trace id tie together?

The gateway, the queue, the engine, and the tool calls in one request. Without it, a slow token cannot be blamed on prefill, a tool, or the GPU.

What does a growing queue mean?

Replicas cannot finish tokens as fast as requests arrive. Adding CPU web workers will not help. You add GPU replicas or shed load.

What do you shed first on an LLM tier?

Anonymous or batch traffic, and requests over a token cap, before you shed the paying interactive path. A full KV cache should reject early.

Where do timeouts sit on a long generation?

A client deadline, a gateway deadline, and a maximum new tokens. A missing cap lets one request hold KV pages until the context limit.

How do you run a nightly completion job?

Off the interactive cluster, with a batch API or a dedicated engine, writing outputs to storage. Sharing the online KV pool makes the product's latency follow the job.

How do several models share a GPU node?

Only with a plan for memory. Two engines that each assume the whole GPU will OOM. The orchestrator assigns devices explicitly so each engine gets the memory it was sized for.

What does tensor parallel demand of the node?

GPUs that can exchange activations every layer, usually NVLink on one machine. Splitting tensor parallel across a slow network makes the step communication-bound.

Why a separate node pool for LLM replicas?

They need specific GPUs, a driver, and a large shared memory. A general pool will schedule them onto the wrong card or evict them for a small CPU pod.

Why pin the NVIDIA driver and CUDA with the image?

The engine's kernels are built for a CUDA range. A node image that moved the driver will fail the replica at startup, often after the scheduler thought it was healthy.

What is a bad readiness check for vLLM?

A path that returns 200 before the weights are loaded. The load balancer will send users to a replica that can only queue them.

The replica is OOM-killed during traffic. What do you change first?

The max sequence and the max batched tokens, or the quantization, so the KV pool fits beside the weights. A bigger machine is the second lever.

KV cache use sits at the cap and tokens per second fall. What is happening?

The scheduler cannot admit more sequences. Raise capacity, shorten contexts, or turn on a cache policy the engine supports, and watch rejections.

Why log prefix-cache hit rate?

A hit skips prefill. If the hit rate collapses after a template change, time to first token will jump even though the GPU looks busy.

How do you canary a weight update?

A fraction of requests, keyed so one user stays on one revision, compared on the SLIs and a fixed prompt set. Keep a conversation on one revision.

What is a generation golden set?

A fixed list of prompts with checks on format, refusal, and a few expected facts, run against the loaded engine before traffic moves.

What do you register for an LLM besides a score?

The weight revision, the engine version, the quantization, the template, and the eval set name. A registry row that only stores a metric cannot be deployed.

What is Weights & Biases for, next to the registry?

The training curves and the comparison of runs. The registry is what production points at. A run id should be written onto the registry entry.

What is TensorBoard still used for?

Local loss and gradient curves during a run. It does not version the artifact the cluster will roll out.

What do you do with a Horovod launch you inherit?

Treat it as an old allreduce wrapper. New training uses the framework's own distributed launcher. Port the job when you next touch it.

Is DeepSpeed the 2026 inference server?

Its training ZeRO path is current. Its inference engine is rarely the one you scale. New serving work starts from vLLM, SGLang, TensorRT-LLM, or Triton.

How do you account for an LLM replica?

GPU-hours times the card, plus tokens out if you need a unit cost. A request count hides that one long generation can cost as much as many short ones.

What has to be true to roll an LLM service back?

The previous weights, tokenizer, template, and engine image are still pullable, and the replica can load them without a rebuild.