vLLM Engine¶

The native vLLM engine runs high-performance LLM inference using vLLM. Unlike the openai_compat engine (which connects to a vLLM server you start yourself), this engine owns the process: the worker launches vllm serve, waits for it to become healthy before claiming jobs, auto-restarts it if it crashes, and stops it on shutdown.

Status¶

Implemented (src/engines/vllm_engine.py).

How it works¶

vllm serve is itself an OpenAI-compatible HTTP server, so VLLMEngine subclasses OpenAICompatEngine and inherits all of its inference, streaming, embeddings, cost-tracking, and model-alias logic. The only thing it adds is subprocess lifecycle management:

Launch — spawns vllm serve <model> --host <host> --port <port> <extra_args> with a merged environment, in its own process group (so the whole vLLM process tree — engine-core + per-GPU workers — can be signalled together).
Health-gated readiness — initialize() does not block on model load. Loading a large model can take minutes (download, quantization, GPU transfer), so the worker returns immediately and health_check() reports False until vLLM's /health endpoint answers 200. The worker only claims jobs for healthy engines, so nothing is routed here until vLLM is ready.
Auto-restart — a background monitor watches the process; on an unexpected exit it relaunches vLLM with exponential backoff (when restart.enabled).
Shutdown — sends SIGTERM to the process group, escalates to SIGKILL after a grace period, and closes the OpenAI client.

vLLM runs in whatever environment launcher resolves to (e.g. a dedicated uv/CUDA venv), so the worker itself does not need vllm, torch, or CUDA installed — only httpx (already a dependency) for the health probe.

Configuration¶

Add vllm to engine.available and configure the engine.vllm block:

engine:
  available:
    - vllm

  vllm:
    launcher: [uv, run, vllm, serve]   # how to invoke vllm (list, or "uv run vllm serve")
    model: openai/gpt-oss-20b          # positional arg passed to `vllm serve`
    host: 127.0.0.1
    port: 8000                         # injected as --port; also builds base_url
    extra_args:                        # appended (do NOT add --host/--port here)
      - --async-scheduling
      - --max-model-len
      - "100000"
    env:                               # merged over the worker's environment
      VLLM_USE_FLASHINFER_SAMPLER: "0"
      CUDA_HOME: /home/jeffreyr/.venv/lib/python3.12/site-packages/nvidia/cu13
    model_aliases:                     # advertised/routed name -> served model name
      gpt-oss-20b: openai/gpt-oss-20b
    health_path: /health               # readiness endpoint polled by the worker
    log_file: ${VLLM_LOG_FILE:-~/.microdc/vllm.log}  # vLLM stdout/stderr destination
    restart:
      enabled: true                    # relaunch vLLM if it exits unexpectedly
      max_backoff: 60                  # cap (seconds) on restart backoff
    concurrency: ${VLLM_CONCURRENCY:-32}  # vLLM batches well — run jobs in parallel
    api_key: NONE
    timeout: 600

The config above is equivalent to running, by hand:

VLLM_USE_FLASHINFER_SAMPLER=0 CUDA_HOME=/path/to/cuda \
  uv run vllm serve openai/gpt-oss-20b \
    --async-scheduling --max-model-len 100000 --host 127.0.0.1 --port 8000

Configuration reference¶

Key	Default	Purpose
`launcher`	`["vllm", "serve"]`	Command prefix. String or list.
`model`	(required)	Model passed as the positional `vllm serve` argument.
`host`	`127.0.0.1`	Bind host; also used to build `base_url`.
`port`	`8000`	Injected as `--port`; also used to build `base_url`.
`extra_args`	`[]`	Extra CLI args (e.g. `--async-scheduling`). String or list.
`env`	`{}`	Env vars merged over `os.environ` for the subprocess.
`model_aliases`	`{}`	Internal name → served model name (routing/heartbeat).
`health_path`	`/health`	Readiness endpoint polled for health.
`log_file`	`~/.microdc/vllm.log`	Where vLLM stdout/stderr is appended.
`restart.enabled`	`true`	Auto-relaunch on unexpected exit.
`restart.max_backoff`	`60`	Max seconds between restart attempts.
`concurrency`	`32`	Max concurrent jobs for this engine. (`config/default.yaml` ships 32 as the recommended GPU-host value; the code falls back to 5 only if the key is unset.)
`api_key`	`NONE`	Sent to vLLM (usually unauthenticated).
`timeout`	`600`	Request timeout (seconds) for inference calls.

Tuning concurrency on GPU hosts¶

vLLM continuously batches in-flight requests, so its real concurrency limit is KV-cache memory, not a fixed request count. The concurrency value in this engine's config only controls how many jobs the worker dispatches into vLLM at once — set it too low and you starve the batcher; the GPU sits idle between bursts even though it has huge spare capacity.

Two caps to raise together¶

Setting	Scope	Effect
`engine.vllm.concurrency`	this engine	Max jobs the worker pushes into vLLM at once.
`resources.max_concurrent_jobs` (`MAX_JOBS`)	global, all engines	Hard cap on the sum across engines — clamps `concurrency` if lower.

Raising engine.vllm.concurrency alone does nothing if the global resources.max_concurrent_jobs is smaller. On a dedicated vLLM GPU host, raise both.

Read the signal from vLLM's own log¶

vLLM prints a stats line every few seconds (to log_file, default ~/.microdc/vllm.log):

Engine 000: Avg generation throughput: 542.4 tokens/s, Running: 2 reqs,
Waiting: 0 reqs, GPU KV cache usage: 1.3%, Prefix cache hit rate: 20.9%

Interpret it like this:

Waiting: 0 + low GPU KV cache usage → vLLM is starved, not saturated. The bottleneck is upstream (worker concurrency / global cap / job arrival rate). Raise the caps. Example: Running: 4, KV cache usage: 0.7% means ~99% of the batching pool is idle — there is enormous headroom.
Waiting persistently > 0 → vLLM now is the limit; requests are queueing. Back off, or scale out to another worker.
GPU KV cache usage near 90% → you're memory-bound. Either lower --max-model-len (each long context reserves KV) or reduce concurrency.

The goal is to push concurrency up until GPU utilization pegs with throughput flattening, or Waiting starts to persist — whichever comes first.

`--max-model-len` vs. concurrency¶

--max-model-len sets the worst-case KV reservation per sequence; vLLM logs Maximum concurrency for <N> tokens per request: X.Xx at startup. With a large value (e.g. 100000) on a 32GB card, that worst-case number is small — but PagedAttention only allocates KV for tokens actually produced, so short requests keep KV usage tiny and concurrency high regardless. Only lower --max-model-len if you genuinely serve long contexts and see KV pressure; don't reduce it just to chase a higher concurrency number.

Efficient tuning procedure¶

Don't increment concurrency by 1 and re-test — that's slow. Use GPU KV cache usage as a linear predictor to jump straight to the right neighborhood, then confirm the knee. Under a representative load (real prompt/output sizes — KV scales with actual tokens, not request count):

Start at the default concurrency: 32 (and MAX_JOBS ≥ that). Let load stabilize and read one stats line: note Running, Waiting, GPU KV cache usage, and Avg generation throughput.
Predict the concurrency that lands near ~80% KV (leaving headroom for prompt spikes / longer contexts):

target_concurrency ≈ current_concurrency × (0.80 / current_KV_fraction)

e.g. 32 reqs at 0.40 KV → 32 × (0.80 / 0.40) ≈ 64. Jump there directly. 3. Confirm the knee at the new value: - Aggregate generation throughput still rising, Waiting: 0, KV < ~85% → headroom remains; nudge up again. - Throughput flat while GPU-util pegs ~100% → compute-bound: this is the ceiling. Set concurrency at or just below this point. - Waiting persistently > 0, or KV → ~90% → past the limit; back off. 4. Settle at the lower of the compute knee and ~85% KV. Keep MAX_JOBS ≥ the final per-engine value so the global cap never silently clamps it.

Two practical accelerators: raise the prefix-cache hit rate by putting shared content (system prompt) first in every request — that cuts prefill work and frees headroom; and Running == concurrency with Waiting == 0 always means you are the cap, not vLLM — keep climbing until that stops being true.

Worked example — RTX 5090 (32 GB), `gpt-oss-20b`¶

Measured on a real worker (mdcw06), short chat-sized requests:

`engine.vllm.concurrency`	`Running`	`Waiting`	KV usage	Agg. gen throughput	Read
5	2–4	0	~1%	~540 tok/s	Badly starved — GPU ~idle between bursts
32	26–32	0	~40%	~1300–1530 tok/s	~3× throughput, but still cap-bound (Running pinned at 32, KV only 40%)
~64 (next step)	—	—	~80% (predicted)	—	Expected near the KV/compute knee

Takeaways: the jump from 5 → 32 tripled throughput purely by un-starving the batcher; 32 was still worker-cap-bound (KV 40%, no queue), so ~64 is the predicted next target via the formula above. Numbers are request-size dependent — re-measure for your own prompt/output lengths rather than copying a value.

Production install (systemd / `ubuntu_setup.sh`)¶

ubuntu_setup.sh --with-vllm provisions a dedicated vLLM environment at /srv/microdcworker/vllm-env (via uv) so the worker can launch it under the hardened systemd unit. In that setup, point launcher directly at the env's binary — this avoids needing uv run (and its cache/project resolution) at runtime:

engine:
  available:
    - vllm
  vllm:
    launcher: [/srv/microdcworker/vllm-env/bin/vllm, serve]
    model: openai/gpt-oss-20b
    port: 8000
    extra_args: [--async-scheduling, --max-model-len, "100000"]
    model_aliases: { gpt-oss-20b: openai/gpt-oss-20b }

The unit sets HOME=/srv/microdcworker and UV_CACHE_DIR under the install dir, so vLLM's caches and the engine's ~/.microdc/vllm.log fall within the unit's ReadWritePaths. On GPU hosts, ensure /dev/shm is large enough for vLLM's workers if you use tensor parallelism.

Sizing a model to VRAM¶

vLLM reserves gpu_memory_utilization (default 0.9) of the GPU at startup; that pool holds weights + KV cache. So usable ≈ 0.9 × VRAM, and KV budget ≈ usable − weights. More KV → more concurrency. Pick a precision so weights leave a healthy KV budget:

Precision	Bytes/param	Qwen3-32B weights	Notes
bf16/fp16	2	~64 GB	Full quality. Needs an 80 GB+ GPU (or tensor-parallel).
FP8	1	~33 GB	Native on Blackwell/Hopper — faster and frees KV. Tiny quality cost.
AWQ/GPTQ 4-bit	0.5	~18–20 GB	Fits a 24–32 GB GPU. Largest KV budget; small quality cost.

Dense vs MoE matters. A dense 32B (Qwen3-32B) loads all 32B params; an MoE like gpt-oss-20b (~3.6B active, MXFP4) is far lighter (~13 GB) despite a similar "size." Don't size by the headline parameter count alone.

Worked examples:

gpt-oss-20b on a 32 GB RTX 5090: MXFP4 weights ~13 GB → ~15 GB KV. Tons of headroom; concurrency is request-size-bound, not memory-bound.
Qwen3-32B on a 96 GB RTX PRO 6000 Blackwell: bf16 weights ~64 GB → ~22 GB KV (~40+ concurrent short requests at 32k ctx). FP8 → ~53 GB KV for roughly double the concurrency, accelerated by Blackwell FP8 tensor cores.

Levers when weights are too big or KV too small:

Quantize (bf16 → FP8 → 4-bit) — each step roughly halves weight memory.
--tensor-parallel-size N to split weights across N GPUs.
--kv-cache-dtype fp8 to roughly double KV capacity.
Lower --max-model-len (only if you don't need the context) — see the concurrency-tuning notes above.

Single model per process¶

A vllm serve process serves exactly one model. This engine launches a single configured model at startup and serves only that model — it does not switch models on demand. To serve a different model, change engine.vllm.model and restart the worker.

Model name mapping¶

Jobs reference the internal name (the left side of model_aliases, e.g. gpt-oss-20b), which the engine translates to the served name vLLM exposes (openai/gpt-oss-20b) on each request. While vLLM is still starting up, list_models() returns an empty list so heartbeats stay quiet until it's ready.

First boot: tracking the model download¶

On first launch vLLM downloads the model weights from Hugging Face into the cache before loading them — for a large model this is the longest part of startup, and the log goes quiet during it (the hf-xet downloader doesn't stream progress bars to a file). The engine is health-gated, so the worker simply claims no jobs until it's ready. Ways to confirm it's progressing:

Watch the HF cache fill (most direct). The service runs with HOME=/srv/microdcworker, so weights land under its cache:

watch -n5 "sudo du -sh /srv/microdcworker/.cache/huggingface/hub/models-* 2>/dev/null"

e.g. models--Qwen--Qwen3-32B. When it stops growing near the model's full size (~65 GB for Qwen3-32B bf16), the download is done. - Watch nvidia-smi — VRAM stays low during download, then jumps to the weights' size the moment loading to GPU begins (i.e. the download finished). - Poll /health — the canonical "ready" signal; returns 200 only after weights load, CUDA-graph capture, and KV profiling all complete:

watch -n5 'curl -s -o /dev/null -w "%{http_code}\n" 127.0.0.1:8000/health'

Watch the log advance past Loading model from scratch... to GPU KV cache size: …, Maximum concurrency for <N> tokens per request: …, and Application startup complete.

Sequence: disk fills (download) → VRAM fills (load to GPU) → GPU-util spikes for 1–3 min (compile/cudagraph) → /health 200.

Tip: set HF_TOKEN in config/environment — unauthenticated HF downloads are rate-limited and slower.

Troubleshooting¶

Engine never becomes healthy — check log_file (default ~/.microdc/vllm.log) for vLLM startup errors (CUDA/driver mismatch, OOM, bad extra_args).
Worker can't find vllm — ensure launcher resolves in the worker's PATH/working directory (e.g. uv run from the right project, or an absolute path to the vLLM venv).
Port already in use — another process (possibly a stale vLLM) holds the port; stop it or change port.
Blackwell (sm_120) needs the cu130 torch build. Root cause of most Blackwell startup failures: vLLM 0.23.0's kernels are built for CUDA 13, but --torch-backend=auto tends to under-pick cu129 (CUDA 12.9) on these cards. That mismatch surfaces as any of: libcudart.so.13: cannot open shared object file (at import), Failed to get device capability: SM 12.x requires CUDA >= 12.9, or RuntimeError: Expected max_shared_mem > 0 ... got false in the Marlin MXFP4 repack (gpt-oss). Confirmed by comparing a working host (torch 2.11.0+cu130) to a broken one (torch 2.11.0+cu129), same vLLM. Fix:

sudo VLLM_PIP_ARGS="--torch-backend=cu130" bash ubuntu_setup.sh   # yes to vllm

Check the installed build with vllm-env/bin/python -c "import torch; print(torch.__version__)" — you want +cu130. ubuntu_setup.sh now auto-detects compute capability ≥ 12 and defaults to cu130, so fresh installs shouldn't hit this. Once on cu130, the LD_LIBRARY_PATH/CUDA_HOME env workaround below is no longer needed. - RuntimeError: The NVIDIA driver on your system is too old (found version 12080) — the flip side of the cu130 requirement: cu130 torch needs a CUDA-13 driver, but this host's driver only exposes CUDA 12.8 (12080). Check with nvidia-smi — the header's CUDA Version: must be 13.0 (driver ≥ 580.x). If it's lower, upgrade the driver to match a known-good host (e.g. sudo ubuntu-drivers install nvidia:580 or sudo apt install -y cuda-drivers-580, then reboot) before reinstalling vLLM on cu130. Note the bind: cu129 torch loads on a 12.8 driver but mismatches vLLM's cu13 kernels; cu130 matches the kernels but needs the newer driver — so a CUDA-13 driver is mandatory for Blackwell + vLLM 0.23.0, not optional. - **Blackwell (sm_120): `Failed to get device capability: SM 12.x requires CUDA

= 12.9** — usually **benign**. vLLM falls back to the FlashAttention v2 backend and serves normally (verified on an RTX PRO 6000 Blackwell running Qwen3-32B). Only act on it if startup actually fails or the first request errors withno kernel image is available for execution— then rebuild the vLLM env against a newer CUDA:sudo VLLM_PIP_ARGS="--torch-backend=cu130" bash ubuntu_setup.sh --with-vllm(theautopick can land on a CUDA < 12.9 torch wheel). TheVLLM_USE_FLASHINFER_SAMPLER=0env (in the example config) is the matching workaround for the FlashInfer sampler on Blackwell. - **import vllm._C → ImportError: libcudart.so.13: cannot open shared object file** — vLLM's compiled extension is built against CUDA 13 but the installed torch/CUDA doesn't provide that runtime lib (an--torch-backend=autopick out of step with the vLLM wheel). It fails at *import*, before any model, so it's independent of the configured model. Diagnose withvllm-env/bin/python -c "import torch; print(torch.version.cuda)"andfind vllm-env -name 'libcudart.so*'. If torch is CUDA 12.x / the lib is absent → rebuild forcing CUDA 13:sudo VLLM_PIP_ARGS="--torch-backend=cu130" bash ubuntu_setup.sh --with-vllm(fallbackcu129). If the lib *is* present but unfound → add its dir toLD_LIBRARY_PATHin the engine'senv:block. A re-run of--with-vllm(rm -rf vllm-env` + reinstall) can flip a previously-working env into this state if dependency resolution changes.