gpu-training-notebooks · serving track

The Serving Console

Pick a GPU, a model and an operating point. This computes what the deployment will actually do — and shows every step of the arithmetic, because a planning model you can't inspect is a rumor. Same equations as the notebooks, ported line for line.

Plan a deployment

roofline decode · KV pool capacity · topology from VRAM · cost

Every output below is derived, never looked up. Change any control and watch which term starts binding — that term, not the brochure, is what your next optimization has to attack.

Weight precision

TP width is chosen for you: the smallest power of two that holds the weights and a KV pool big enough for 32 concurrent requests.

the arithmetic, in order
where this config sits on the roofline — ridge point ● achieved · ○ ceiling

Triage a running engine

paste vLLM stats lines · get the cause, not the symptom

One periodic log line is most of production debugging. The rules below are the ones from the observability notebook, unchanged: they alert on causes — KV pressure, queue depth, prefill starvation — rather than on the latency those causes produce.

Tolerant parser: missing fields are fine, and a line it can't read is skipped rather than crashing.