Inference stack

What does vLLM add around the weights?

Paged storage for the KV cache, continuous batching of decode steps, and an OpenAI-compatible HTTP server. The Python API is a client of that runtime.

What problem does paged KV storage solve in vLLM?

The cache for each sequence grows as tokens are generated. Fixed reservations waste GPU memory. Pages let the server pack many sequences into the memory it actually has.

What does continuous batching change about a step?

A finished sequence leaves the batch and a waiting one joins at the next token. The GPU does not sit idle until the longest request in the batch ends.

What is prefix caching on a server?

Shared prompt prefixes keep their KV pages and later requests reuse them. A fixed header, as in an agent loop, benefits. A fully unique prompt misses the cache.

What is SGLang built to serve?

High-throughput generation with a radix cache of shared prefixes and fast constrained decoding. It is a server, in the same job category as vLLM, with a different scheduler.

What is a radix cache?

A tree of shared token prefixes. Two requests that share a header reuse the computed KV for that header instead of prefilling it twice.

How do you choose between vLLM and SGLang?

Measure time to first token and tokens per second on your prompts, including structured outputs if you use them. Report the prompt mix with the number.

What is constrained decoding on a server?

The sampler is limited to tokens that still match a grammar or a JSON schema. Invalid JSON is rejected during generation, not cleaned up afterward.

What does llama-cpp-python bind to?

llama.cpp, a C++ runtime that loads GGUF files and can offload layers to a GPU. It is the right local runner. It is not the scheduler for a multi-tenant cluster.

What is a GGUF file?

One file of quantized weights plus metadata that llama.cpp reads. The quantization method, such as Q4_K_M, is chosen when the file is built, not at request time.

What does the Ollama Python client talk to?

A local daemon that has pulled a model and exposes a small HTTP API. The daemon picks the runner. You do not set the server's batching policy from that client.

When is Ollama the right runtime?

A developer machine or a single-user tool, where a daemon and a model name are enough. A service with a queue, an SLO, and several replicas wants a dedicated engine.

What is mlx-lm for?

Inference and fine-tuning on Apple Silicon, using unified memory. It is the Mac path. A CUDA server does not load an MLX checkpoint.

Why can a Mac hold a model that a small GPU cannot?

Unified memory is shared by the CPU and the GPU, so the weights do not have to be copied into a separate GPU heap. Bandwidth, not the CUDA kernel set, is the limit.

What is an ONNX file?

A frozen graph of ops and weights, exported from the training framework. The runtime executes that graph. A new control-flow trick in PyTorch needs a new export.

What does ONNX Runtime add to the file?

Execution providers for CPU, CUDA, or TensorRT, chosen at session creation. The same file can run on more than one provider, at different speeds.

What breaks an ONNX export of a language model?

Sequence length and batch size that were fixed at export. Mark those axes as dynamic, then test a longer prompt than the one you exported with.

What does Optimum do in the path?

It exports a Hugging Face model to ONNX, OpenVINO, or a TensorRT path and loads it back. It is the glue. It is not the server that queues users.

What does TensorRT-LLM build?

An engine for one GPU architecture and one precision. The build is slow and the engine does not move unchanged to a different GPU generation.

When is the TensorRT-LLM build worth it?

When the model and the GPU are stable and the request rate pays for the engine. A checkpoint that changes daily spends its life rebuilding.

What is ExLlamaV2 for?

Fast inference of EXL2 quantized models on a single NVIDIA GPU or a small box. It targets local and single-tenant speed, not a cluster scheduler.

What is LMDeploy?

A serving stack, with the TurboMind runtime, that exposes an API similar to the other LLM servers. You benchmark it as a peer of vLLM, not as a Python generate() loop.

What is Llama Stack?

An API layer for Llama-oriented inference and agent calls. It sits above a runtime. It does not implement the attention kernel.

What does the flash-attn package provide?

An attention implementation that tiles the softmax so the full attention matrix is not stored. It must be built for your PyTorch and CUDA versions or the import fails.

When is PyTorch SDPA enough?

When its scaled-dot-product attention already uses a flash or memory-efficient kernel for your shapes. Use the flash-attn package for a feature that SDPA build lacks.

What is the Triton language?

A Python-like compiler for GPU tile programs. Attention kernels are often written in it. The NVIDIA inference server is a different project that shares the name.

When do you write a Triton kernel?

When one PyTorch op dominates the step and a fused tile would cut memory traffic. You measure that op first. A new kernel for a cheap op costs more than it saves.

What does torchao quantize?

Weights inside the PyTorch eager or compile path, including int4, int8, and fp8 layouts. You stay in PyTorch. You do not get a TensorRT engine.

How does torchao differ from bitsandbytes?

bitsandbytes is the usual 4-bit load and 8-bit optimizer during training. torchao is PyTorch's own quantization path for inference and for some training layouts.

Where do GPTQ and AWQ live in 2026?

The methods are still how post-training integer weights are built. The older auto-gptq and autoawq repos are often behind the loaders inside vLLM and transformers.

Why can't a QLoRA training save be served as GGUF?

NF4 bitsandbytes weights are a training representation. GGUF, AWQ, and FP8 are serving files. You convert or requantize, then score the converted file.

What is FP8 serving?

Weights and sometimes activations in 8-bit floating point on GPUs that have those tensor cores. It needs that hardware. A CPU runner will not execute it.

What is speculative decoding on a server?

A small draft model proposes several tokens and the target model checks them in one pass. It helps when the draft is cheap and most proposed tokens are accepted.

What is chunked prefill?

A long prompt is processed in chunks so decode requests already in flight are not blocked for the whole prefill. Prefill and decode then share the GPU by design.

What does an OpenAI-compatible endpoint buy you?

Clients written for chat completions keep working when you change servers. Sampling defaults and error bodies can still differ. Test the ones you rely on.

What is multi-LoRA serving?

One base model stays on the GPU and several adapters are selected per request. The base is loaded once. Each adapter has to be the format that server can apply.

When does a server use tensor parallel?

When the weights do not fit on one GPU. Layers are split across GPUs and those GPUs exchange activations every layer, so they need a fast link.

What are CUDA graphs used for in a decode loop?

The kernel launches of one decode step are captured and replayed. Prefill and decode are captured separately because their shapes differ.

Why exclude the first requests from a latency number?

The server may compile, load adapters, or capture graphs on those calls. Steady latency is the number after that work is done.

A serving GPU runs out of memory after the weights fit. What grew?

The KV cache, with batch size and sequence length. Lower the maximum batched tokens or the maximum sequence, or quantize the cache.

What does max batched tokens control?

How much prefill and decode work the scheduler will put on the GPU at once. Higher raises throughput until the cache or the latency budget breaks.

What is time to first token a measure of?

How long the user waits before any text appears. Prefill length, queueing, and prefix-cache hits move it. Decode speed moves the tokens after that.

What must a server do when a client disconnects?

Free that sequence's KV pages and stop its decode. A server that finishes every disconnected request wastes the GPU.

Why do flash-attn installs fail on a new PyTorch?

The wheel is compiled for a specific PyTorch and CUDA pair. An install that looks successful can still fail at import when those do not match.

What does a Python inference package usually contain?

A client and a thin binding. The paging, the kernels, and the scheduler are in a compiled runtime. Reading only the Python will not show the memory layout.

When do you skip the online server?

When the inputs are a file and nobody is waiting on a token. A batch job that writes completions can use a simpler runner and a bigger batch.

What should a model server's health check load?

That the weights are resident and a one-token forward pass works. A check that only proves the HTTP port is open will pass a server that cannot generate.

Which numbers does the server need to expose?

Queue time, time to first token, tokens per second, KV cache use, and the fraction of requests rejected. GPU utilization alone hides a full cache.

What is a practical 2026 serving choice on NVIDIA GPUs?

vLLM or SGLang behind the API your clients already speak, with the quantization format that engine's kernels implement, and a measured cap on batched tokens.