Serve an LLM with vLLM on CPU in Docker: Learn the Production Engine Without a GPU (2026)

vLLM is the serving engine the GPU crowd reaches for. It grew out of UC Berkeley's PagedAttention research — the SOSP 2023 paper that treats the KV cache like virtual memory — has been hosted by the PyTorch Foundation since May 2025, and sits at ★86.9k GitHub stars (86,875 — RECATOOLS traction data, 22 Jul 2026). Its README leads with "State-of-the-art serving throughput". Its CPU documentation uses a much quieter phrase: "basic model inferencing and serving". Both are the project's own words. The gap between them is the premise of this tutorial.

So why run a GPU-first engine on a CPU at all? Because the parts of vLLM that make it a production system — the OpenAI-compatible API, the Prometheus /metrics endpoint, the engine flags, the continuous-batching configuration — are exactly the same in CPU mode, and CPU mode runs on the laptop you already own. You are not here for speed, but to learn the production surface before you rent the GPU. Our vLLM directory entry puts it plainly: a single user wanting one model on a laptop is better served by the llama.cpp family — vLLM earns its complexity when there are concurrent requests to batch. Nothing below contradicts that.

We ran every command in this tutorial on 22 July 2026. The testbed was Docker Desktop for Apple Silicon: a Linux VM with 6 CPUs and 7.75 GiB of RAM, with our dev-stack containers running alongside. We pinned vLLM v0.25.1 (released 14 July 2026), served Qwen/Qwen3-0.6B through the official CPU image, built the same image from source, and hit three traps on the way. All three are reproducible, and one of them took Docker Desktop down with it.

The fork in the road: which image?

Two vLLM Docker Hub repositories matter here, and the wrong one looks right. The flagship vllm/vllm-openai image is the CUDA build — and its v0.25.1 manifest includes a linux/arm64 entry, so on an M-series Mac a docker pull would succeed without complaint. We checked the manifest instead of downloading 10 GB to find out:

docker manifest inspect vllm/vllm-openai:v0.25.1 | grep -A3 platform
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
--
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }

That arm64 entry is not for your Mac. The project's CUDA docs explain who it is for: "A docker container can be built for aarch64 systems such as the Nvidia Grace-Hopper and Grace-Blackwell" — data-center machines that pair an ARM CPU with an NVIDIA GPU. It is a ~10.2 GB compressed download (the amd64 variant is ~8.8 GB) that expects a CUDA driver your Docker VM does not have. We did not run it, so we will not tell you what error you would get; we will just tell you not to spend the 10 GB finding out.

The image you actually want is vllm/vllm-openai-cpu — official, current (last updated 13 July 2026, 144,916 pulls), and documented on the CPU installation page with this exact pull pattern. The arm64 tag is 857 MB compressed; the x86_64 tag is 1.76 GB.

# From docs.vllm.ai, CPU installation (ARM tab) — the documented pull commands
docker pull vllm/vllm-openai-cpu:latest-arm64
docker pull vllm/vllm-openai-cpu:v${VLLM_VERSION}-arm64   # pin the version; we used v0.25.1

The docs can seem contradictory on one point: the Apple-silicon tab says "Currently, there are no pre-built Arm silicon CPU images." That sentence is about running vLLM natively on macOS, which is a separate, experimental, build-from-source path. Docker Desktop runs Linux containers, and for Linux arm64 the pre-built image above exists and works. Same laptop, different operating system inside the box.

The quick path: pulling it (three times, honestly)

The pull took three tries on our machine, for reasons that had nothing to do with vLLM. The first died mid-download on a transient network error:

docker pull vllm/vllm-openai-cpu:v0.25.1-arm64
v0.25.1-arm64: Pulling from vllm/vllm-openai-cpu
d98d9a7c42e7: Pulling fs layer
…
failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://production.cloudfront.docker.com/registry-v2/docker/registry/v2/blobs/sha256/4a/…": net/http: TLS handshake timeout

The second attempt was fired at a daemon that turned out to be down — Docker Desktop was showing "Docker Desktop is unable to start" at the time. That was the aftermath of the source-build crash that is a story for later; the retry simply ran into it. The CLI's response is a 502 with a mildly unhinged URL:

docker pull vllm/vllm-openai-cpu:v0.25.1-arm64   # retry — first attempt hit a TLS handshake timeout
request returned 502 Bad Gateway for API route and version http://…docker.sock/v1.55/images/create?fromImage=docker.io%2Fvllm%2Fvllm-openai-cpu&tag=v0.25.1-arm64, check if the server supports the requested API version

After a quit-and-relaunch of Docker Desktop, attempt three ran clean:

docker pull vllm/vllm-openai-cpu:v0.25.1-arm64
v0.25.1-arm64: Pulling from vllm/vllm-openai-cpu
…
Digest: sha256:939aea76a6a3b7ea4a309084b7ba437e61656ffd26817b45130483ed55d830a5
Status: Downloaded newer image for vllm/vllm-openai-cpu:v0.25.1-arm64
docker.io/vllm/vllm-openai-cpu:v0.25.1-arm64

The 857 MB figure is Docker Hub's compressed size; unpacked on disk the image is 3.58 GB. Budget for the larger number.

Trap #3: the memory flag named for a GPU that isn't there

A quick housekeeping note on numbering: we number the traps in the order our session hit them, and our session built from source before it pulled. On the quick path, the trap you meet first is #3.

The run command below is the docs' own recipe for the ARM image: --security-opt seccomp=unconfined --cap-add SYS_NICE because Docker's default seccomp profile blocks the NUMA syscalls vLLM uses for memory binding (the docs note functionality is unaffected without them, but performance can be), --shm-size=4g from the same documented example, and a named volume on ~/.cache/huggingface so the model survives container restarts. The model argument goes straight through to vllm serve — the image's entrypoint is exactly that command. We picked Qwen/Qwen3-0.6B: Apache-2.0, ungated, a ~1.5 GB download from Hugging Face on first run, and the same model vLLM's own docs use in their examples. Regular readers have met these weights before; hold that thought until the Merlion shows up.

lsof -nP -iTCP:8000 -sTCP:LISTEN
(no output — port 8000 is free)
docker run -d --name vllm-cpu-box --security-opt seccomp=unconfined --cap-add SYS_NICE --shm-size=4g -p 8000:8000 -v vllm-hf-cache:/root/.cache/huggingface vllm/vllm-openai-cpu:v0.25.1-arm64 Qwen/Qwen3-0.6B --dtype=bfloat16 --max-model-len 8192
2a139360807c1c4df1834de2946e4917978891a7294fe3ecd56482607a4bbd85

The container came up, downloaded the model — and then the engine worker died during initialization. The API server's log ends in a generic failure:

docker logs vllm-cpu-box 2>&1 | tail -40
…
(APIServer pid=1)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {'EngineCore': 1}

The root cause, further up the log, is an unusually helpful error message. It diagnoses the trap, explains the misnamed flag, and gives you the fix:

ValueError: Available memory on node 0 (4.66/7.75 GiB) on startup is less than desired CPU memory utilization (0.92, 7.13 GiB). On the CPU backend, the `--gpu-memory-utilization` flag controls the fraction of CPU memory reserved (despite its name). To resolve: decrease `--gpu-memory-utilization` (e.g. `--gpu-memory-utilization 0.5`) or reduce CPU memory used by other processes.

This is the key to vLLM's CPU mode: the --gpu-memory-utilization flag governs CPU memory. Its default of 0.92 reserves 92% of the VM — 7.13 of our 7.75 GiB. Our VM had 4.66 GiB free because the site's dev stack was running alongside. A fresh VM will have more headroom, but the default still claims nearly all of it, so any other resident container puts you exactly here. As instructed:

docker rm -f vllm-cpu-box
vllm-cpu-box
docker run -d --name vllm-cpu-box --security-opt seccomp=unconfined --cap-add SYS_NICE --shm-size=4g -p 8000:8000 -v vllm-hf-cache:/root/.cache/huggingface vllm/vllm-openai-cpu:v0.25.1-arm64 Qwen/Qwen3-0.6B --dtype=bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.5
2665a5abf5bd07e8343179797aaf66365725ed7d465984ee415e261ca35ebd99

What a healthy CPU startup looks like

This time the engine came up. Five log lines tell the whole story, including one that keeps the flag's joke running:

docker logs vllm-cpu-box 2>&1 | grep -E "KV cache|Available KV|init engine|startup complete|Uvicorn|chunked prefill|torch_dtype|Loading model|loaded in"
(Worker pid=75) INFO 07-22 15:49:14 [selector.py:138] Using HND KV cache layout for CPU_ATTN backend.
(Worker pid=75) INFO 07-22 15:50:44 [cpu_worker.py:235] Auto set (1.05/7.75) GiB for KV cache on node 0, with 3.88 GiB requested memory for the worker. 2.83 GiB memory was consumed by non-kv usages.
(EngineCore pid=50) INFO 07-22 15:50:44 [kv_cache_utils.py:2146] GPU KV cache size: 9,728 tokens
(EngineCore pid=50) INFO 07-22 15:50:45 [core.py:344] init engine (profile, create kv cache, warmup model) took 29.29 s
(APIServer pid=1) INFO:     Application startup complete.

Yes, that third line reads "GPU KV cache size: 9,728 tokens" on a machine with no GPU. The CPU backend borrows the GPU code paths' vocabulary wholesale; once you know that, the misnamed flag and log line make a certain kind of sense.

The second line is the important one: within our halved memory budget, the worker auto-sized a 1.05 GiB KV cache, enough for 9,728 tokens of context across concurrent requests. The documented lever for this is the VLLM_CPU_KVCACHE_SPACE environment variable — set it to n to reserve n GiB for the KV cache; left unset, vLLM's source resolves it to a 4 GiB reservation (the docs' FAQ: "This value is 4GB by default"). Its sibling VLLM_CPU_OMP_THREADS_BIND pins OpenMP threads to cores and defaults to auto. On a serious CPU host — a Graviton or Xeon box — these two are your main tuning surface. In a 7.75 GiB VM, the auto-sizer's answer is the right one.

The payoff: a real OpenAI-compatible API on port 8000

Everything from here on is the production surface, verbatim. First, the model listing:

curl -s http://localhost:8000/v1/models | python3 -m json.tool
{
    "object": "list",
    "data": [
        {
            "id": "Qwen/Qwen3-0.6B",
            "object": "model",
            "created": 1784735483,
            "owned_by": "vllm",
            "root": "Qwen/Qwen3-0.6B",
            "parent": null,
            "max_model_len": 8192,
…

Any OpenAI client library pointed at http://localhost:8000/v1 will now talk to this server. Then the question this guide series always asks, and the answer that earns its place in the transcript:

curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"What is the Merlion? Answer in two sentences."}],"max_tokens":1024}' | python3 -m json.tool
{
    "id": "chatcmpl-8ff08df13667f406",
    "object": "chat.completion",
    "created": 1784735488,
    "model": "Qwen/Qwen3-0.6B",
…
                "content": "<think>\nOkay, the user is asking about the Merlion and wants the answer in two sentences. First, I need to recall what the Merlion is. From what I remember, the Merlion is a large, ancient lion, also known as the lion of the Nile. It's a mythical creature from ancient Egyptian mythology, often associated with the Nile River and the god of the Nile, which is also the god of the sky. So, the first sentence should mention the name and its connection to the Nile. The second sentence needs to explain its significance in mythology and its role in ancient Egyptian culture. Let me check if I got that right. Yes, the Merlion is a lion from ancient Egyptian mythology, symbolizing power and the Nile's role in life and death. That covers both the name and its mythological context. I think that's correct.\n</think>\n\nThe Merlion is a large, ancient lion from ancient Egyptian mythology, symbolizing power and the Nile River's role in life and death. It represents the god of the Nile, who governs the land and the sky.  \n\nThe Merlion is a legendary creature believed to have appeared in the Nile's waters, embodying the balance between the earth and the heavens, and is often associated with themes of strength and wisdom in ancient Egyptian culture.",
…
            "finish_reason": "stop",
…
    "system_fingerprint": "vllm-0.25.1-f9d194c9",
    "usage": {
        "prompt_tokens": 19,
        "total_tokens": 285,
        "completion_tokens": 266,
…
curl -s http://localhost:8000/v1/chat/completions -H  -d   0.00s user 0.00s system 0% cpu 21.254 total
python3 -m json.tool  0.03s user 0.01s system 0% cpu 21.260 total

The Merlion — Singapore's half-lion, half-fish harbor statue — is now "a large, ancient lion from ancient Egyptian mythology", "the lion of the Nile", an emissary of "the god of the Nile". The response's own think block is the most instructive part: you can watch the model confidently "recalling" facts that never existed ("From what I remember…", "Let me check if I got that right. Yes…"). Self-checks inside an ungrounded model check against the same wrong memory.

Regular readers know this is the third round of a running experiment. In our Ollama-on-Mac guide, these same qwen3:0.6b weights relocated the Merlion to New York. In the AnythingLLM tutorial, RAG grounding finally pinned it to Singapore. On 22 July the identical weights, served by a production-grade engine, invented Egypt. The serving stack changed; the model didn't. Infrastructure buys you throughput and observability. Grounding buys you facts.

On speed: dividing the 266 completion tokens by the 21.254-second wall clock gives about 12.5 tokens per second. That is a figure derived from one transcript on one loaded VM, not a benchmark — we ran no other timing — but it sets expectations correctly: CPU vLLM in a small VM is interactive-demo speed, not production speed.

The observability payoff: /metrics

The docs' claim is that metrics "are exposed via the /metrics endpoint on the vLLM OpenAI compatible API server", Prometheus-compatible, under a vllm: prefix. Verified — and the counters reconcile exactly with the usage block above:

curl -s http://localhost:8000/metrics | grep -E "^vllm:(num_requests|prompt_tokens|generation_tokens|request_success)" | head -8
vllm:num_requests_running{engine="0",model_name="Qwen/Qwen3-0.6B"} 0.0
vllm:num_requests_waiting{engine="0",model_name="Qwen/Qwen3-0.6B"} 0.0
…
vllm:prompt_tokens_total{engine="0",model_name="Qwen/Qwen3-0.6B"} 19.0
…

prompt_tokens_total reads 19.0; the chat response reported "prompt_tokens": 19. One request in, one request accounted for. This endpoint is how production vLLM deployments are watched, scraped by Prometheus and graphed in Grafana. It works identically on a laptop CPU.

The build-from-source path — and traps #1 and #2

Most readers should stop at the pulled image. Build from source when you need a patched engine, a custom dependency set, or an image for hardware the published tags don't cover. It is also where our session started, and where Docker Desktop's one truly bad moment of the session happened. Clone at the tag:

git clone --depth 1 --branch v0.25.1 https://github.com/vllm-project/vllm.git
Cloning into 'vllm'...
Note: switching to '752a3a504485790a2e8491cacbb35c137339ad34'.
…
cd vllm && git log --oneline -1
752a3a5 [Bugfix] Guard mixed-dtype allreduce RMSNorm quant fusions (#48330)

Trap #1: the default build parallelism kills the VM

The repo ships docker/Dockerfile.cpu, whose header states it "is used to build images that can run vLLM on both x86_64 and arm64 CPU platforms." Line 29 of the same file defaults max_jobs=32 — a compile parallelism sized for CI hardware, not for a 6-CPU, 7.75 GiB Docker VM. We ran the naive build anyway, to see what a reader would see:

docker build -f docker/Dockerfile.cpu --progress=plain -t vllm-cpu:v0.25.1 .
#0 building with "desktop-linux" instance using docker driver
…
#35 350.1 [64/279] Building CXX object /vllm-workspace/.deps/onednn-build/src/cpu/CMakeFiles/dnnl_cpu.dir/float16.cpp.o
ERROR: failed to build: failed to receive status: rpc error: code = Unavailable desc = error reading from server: EOF
EXIT_CODE=1

That EOF is BuildKit losing its server mid-compile, about six minutes in, during the VLLM_TARGET_DEVICE=cpu wheel-build stage. The VM went down at the default 32-way parallelism — Docker Desktop then reported "Docker Desktop is unable to start" and needed a full quit-and-relaunch before anything else would run. We did not capture a kernel-level diagnosis of why the VM died, so we won't assert one; what we can say is that the crash arrived under a 32-job compile on a 6-CPU/7.75 GiB VM, and that capping the parallelism made it never happen again. The cap we used is not our invention: vLLM's own docs use max_jobs=4 in their cross-compile example.

docker build -f docker/Dockerfile.cpu --build-arg max_jobs=4 --progress=plain -t vllm-cpu:v0.25.1 .
…
#35 4.873 -- ARMv8 or later architecture detected
#35 4.873 -- BF16 extension detected
#35 4.877 -- Downloading Arm Compute Library (ACL) from GitHub
…
#35 5.679 [1/279] Building CXX object /vllm-workspace/.deps/onednn-build/src/common/CMakeFiles/dnnl_common.dir/bfloat16.cpp.o
…
#35 962.9 [279/279] Linking CXX shared module _C.abi3.so
#35 964.2 creating 'dist/vllm-0.25.1+cpu-cp38-abi3-linux_aarch64.whl' and adding 'build/bdist.linux-aarch64/wheel' to it

At four jobs the wheel stage ran to completion — 279 oneDNN objects compiled and linked, with the stage's own clock reading a little over 16 minutes at link time. The aarch64 path also clones the Arm Compute Library inside the build, a roughly 757 MiB in-build git clone we watched come down during the first build; later builds reuse the cached copy. This is not a small build; it is merely, at the right parallelism, a survivable one.

Trap #2: the phantom default target

Wheel built, dependencies installed — and then the build failed anyway, at the very last stage:

…
 > [vllm-openai-zen 1/2] RUN if [ "arm64" != "amd64" ]; then         echo "ERROR: vllm-openai-amd only supports --platform=linux/amd64";         exit 1;     fi:
------
Dockerfile.cpu:315
--------------------
 314 |     
 315 | >>> RUN if [ "$TARGETARCH" != "amd64" ]; then \
 316 | >>>         echo "ERROR: vllm-openai-amd only supports --platform=linux/amd64"; \
 317 | >>>         exit 1; \
 318 | >>>     fi
 319 |     
--------------------
ERROR: failed to build: failed to solve: process "/bin/sh -c if [ \"$TARGETARCH\" != \"amd64\" ]; then         echo \"ERROR: vllm-openai-amd only supports --platform=linux/amd64\";         exit 1;     fi" did not complete successfully: exit code: 1
EXIT_CODE=1

The Dockerfile's header comment lists its build targets, annotating one of them "vllm-openai (default): used for serving deployment". That "default" is the phantom: pass no --target and Docker does what Docker always does — builds the last stage in the file, which here is vllm-openai-zen, an AMD-specific variant whose guard refuses any platform but linux/amd64 (its error string calls itself "vllm-openai-amd"). The header's default is only true if you make it true. The docs' own documented ARM build command includes --target vllm-openai; ours had dropped it. Restored:

docker build -f docker/Dockerfile.cpu --build-arg max_jobs=4 --target vllm-openai --progress=plain -t vllm-cpu:v0.25.1 .   # retry: without --target, docker builds the LAST stage (vllm-openai-zen), which is amd64-only
…
#39 exporting manifest list sha256:fab24a7555ced9d3758ae3b4ad7e18991151a086bf8f07903df1b7c0dedb5005 done
#39 naming to docker.io/library/vllm-cpu:v0.25.1 done
#39 unpacking to docker.io/library/vllm-cpu:v0.25.1
#39 unpacking to docker.io/library/vllm-cpu:v0.25.1 8.5s done
#39 DONE 34.7s
EXIT_CODE=0

For reference, the docs' full documented ARM build command — the one to start from so you never meet trap #2 at all:

# From docs.vllm.ai, CPU installation (ARM tab) — the documented build command
docker build -f docker/Dockerfile.cpu \
    --platform=linux/arm64 \
    --build-arg VLLM_CPU_ARM_BF16=false \
    --tag vllm-cpu-env \
    --target vllm-openai .

Does the home build match the official image?

Two images now sit side by side — ours and theirs, near-identical in size:

docker images | grep vllm
vllm-cpu:v0.25.1  3.6GB
vllm/vllm-openai-cpu:v0.25.1-arm64  3.58GB

We swapped the source-built image into the exact run command from earlier and checked the surface:

docker rm -f vllm-cpu-box
vllm-cpu-box
docker run -d --name vllm-cpu-built --security-opt seccomp=unconfined --cap-add SYS_NICE --shm-size=4g -p 8000:8000 -v vllm-hf-cache:/root/.cache/huggingface vllm-cpu:v0.25.1 Qwen/Qwen3-0.6B --dtype=bfloat16 --max-model-len 8192 --gpu-memory-utilization 0.5
3554d565a26e75ea87867b1879c253063d672d2dc4322f33e23ab48ed92583bd
curl -s http://localhost:8000/v1/models | python3 -c "import json,sys; d=json.load(sys.stdin); print(d[\"data\"][0][\"id\"], d[\"data\"][0][\"max_model_len\"])"
Qwen/Qwen3-0.6B 8192
docker logs vllm-cpu-built 2>&1 | grep "init engine"
(EngineCore pid=49) INFO 07-22 15:53:09 [core.py:344] init engine (profile, create kv cache, warmup model) took 29.29 s

Same model id, same context length — and, in a coincidence we double-checked because we didn't believe it either, the same engine-init time to the hundredth of a second: 29.29 s in both logs, both captured in the transcripts. The builds are equivalent where it matters; the stopwatch agreeing to two decimal places is luck, and we report it only because it happened.

Teardown

Containers-only from start to finish — nothing was installed on the Mac itself, so removal is complete and verifiable:

docker rm -f vllm-cpu-built
vllm-cpu-built
docker volume rm vllm-hf-cache
vllm-hf-cache
docker rmi vllm-cpu:v0.25.1 vllm/vllm-openai-cpu:v0.25.1-arm64
Untagged: vllm-cpu:v0.25.1
Deleted: sha256:fab24a7555ced9d3758ae3b4ad7e18991151a086bf8f07903df1b7c0dedb5005
Untagged: vllm/vllm-openai-cpu:v0.25.1-arm64
Deleted: sha256:939aea76a6a3b7ea4a309084b7ba437e61656ffd26817b45130483ed55d830a5
docker ps -a --filter name=vllm --format "{{.Names}}" ; docker volume ls -q | grep -c vllm ; docker images | grep -c vllm
0
0
(all zero/empty — teardown verified)

That removes both images, the model cache volume (with the 1.5 GB of Qwen weights), and every container. Keep the vllm-hf-cache volume if you plan to come back — it is the only thing worth preserving.

What CPU mode taught us — and what it can't

Tally what transferred. After one afternoon on a laptop you have operated vLLM's real production surface: the OpenAI-compatible endpoints that any client SDK can hit, the engine flags (--dtype, --max-model-len, and yes, --gpu-memory-utilization), the KV-cache sizing behavior that PagedAttention's continuous batching depends on, and a Prometheus metrics endpoint whose counters you have reconciled against a live request. Every one of those skills moves unchanged to a GPU deployment, where the engine's actual reputation was earned.

Performance did not transfer, and the project never claimed it would. "Basic model inferencing and serving" is the CPU pages' own ceiling, and our derived 12.5 tok/s is what basic looks like in a small VM. So place vLLM honestly among its neighbours. If you are one person who wants one model running on a laptop, Ollama and the llama.cpp family remain the right answer — our Ollama guide gets you there in a fraction of the steps, and our self-hosting model roundup covers what to run on it. If you are choosing a serving engine for concurrent traffic, vLLM is what the complexity is for — and once it is up, a gateway like LiteLLM slots in front of it as one OpenAI-format endpoint over many backends, exactly as we wired in the LiteLLM tutorial; a UI like Open WebUI can point at either. And if this run-the-GPU-thing-on-CPU genre appeals, the ComfyUI-on-CPU tutorial is this article's sibling with images instead of tokens.

FAQ

Is CPU serving officially supported, or a hack?

Official, with modest billing: the x86 docs say "vLLM supports basic model inferencing and serving on x86 CPU platform, with data types FP32, FP16 and BF16", and the ARM page mirrors it (NEON required). Pre-built Linux wheels exist for both architectures, and vllm/vllm-openai-cpu is an official Docker Hub image. Native macOS (no Docker) is the experimental tier — "users must build from source" — which is why this tutorial stays inside Linux containers.

Why is the memory flag called `--gpu-memory-utilization` on a CPU backend?

Because the CPU backend reuses the GPU code paths' configuration surface. The error message owns it: "the --gpu-memory-utilization flag controls the fraction of CPU memory reserved (despite its name)". Its default reserves 92% of the machine's memory, which a VM running anything else cannot give — set it explicitly (we used 0.5). The startup log even reports "GPU KV cache size" on a GPU-less box. Read "GPU" as "accelerator budget" throughout and the CPU backend makes sense.

Should I serve models with vLLM or Ollama on my laptop?

For using a model: Ollama. It is the easiest local runtime, it manages quantized models for exactly this hardware, and our directory scores reflect that division of labour. Use vLLM-on-CPU to learn and smoke-test the serving stack you'll later run on GPUs. The payoff is API and operational fidelity, not tokens per second. One data point of ours, derived, not benchmarked: 266 tokens in 21.254 s, about 12.5 tok/s, for a 0.6B model.

Does this work on an Intel/AMD machine too?

Yes — arguably better. Pull vllm/vllm-openai-cpu:v0.25.1-x86_64 (1.76 GB compressed); the docs recommend CPUs with avx512f (avx2 works with limited features), and x86 is where the CPU backend's extras live: the experimental AMX-based VLLM_CPU_SGL_KERNEL path is x86-only, and the CPU quantization options sit there too (AWQ and GPTQ on x86; INT8 W8A8 on x86 and s390x) — none are listed for ARM. The run command is the same, with the -x86_64 tag in place of -arm64.