GPU deployment¶
How to get a device through to LocalAI in Docker, Compose and Kubernetes, and how to prove it is actually being used. The mechanics of image variants, backends and layer offloading are in LocalAI GPU; this page is about the deployment plumbing around them.
Only LocalAI needs a GPU. LocalAGI and LocalRecall are orchestration and I/O; they never touch a device. That single fact is the strongest argument for splitting LocalAI out of an integrated deployment — it is the only component with special hardware requirements.
Three things must agree¶
flowchart LR
T["image tag<br/>-gpu-nvidia-cuda-12"] --> A["accelerated"]
D["device passthrough<br/>--gpus / device plugin"] --> A
B["backend variant<br/>chosen at model install"] --> A
LocalAI does not warn when they disagree. It installs a CPU backend and runs.
Docker¶
docker run -d --name localai -p 8080:8080 --gpus all \
-v localai-models:/models -v localai-backends:/backends \
localai/localai:v4.8.2-gpu-nvidia-cuda-12 qwen3-1.7b
Prove the runtime can reach the device before blaming LocalAI:
If that fails, the NVIDIA Container Toolkit is the problem.
| Vendor | Tag suffix | Passthrough |
|---|---|---|
| NVIDIA | -gpu-nvidia-cuda-12, -gpu-nvidia-cuda-13 |
--gpus all |
| AMD | -gpu-hipblas |
--device /dev/kfd --device /dev/dri |
| Intel | -gpu-intel |
--device /dev/dri |
| Any | -gpu-vulkan |
--device /dev/dri |
| Jetson | -nvidia-l4t-arm64, -nvidia-l4t-arm64-cuda-13 |
--runtime nvidia |
Compose¶
services:
localai:
image: localai/localai:v4.8.2-gpu-nvidia-cuda-12
ports:
- "8080:8080"
volumes:
- localai-models:/models
- localai-backends:/backends
- localai-configuration:/configuration
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
interval: 15s
retries: 120
start_period: 30s
For AMD or Intel, replace the deploy.resources block with device mappings:
Everything else in the reference environment is unchanged — LocalAGI, LocalRecall and PostgreSQL are unaffected by the switch.
Kubernetes¶
spec:
containers:
- name: localai
image: localai/localai:v4.8.2-gpu-nvidia-cuda-12
resources:
limits:
nvidia.com/gpu: 1
A GPU is a non-overcommittable resource: limits must equal requests, and one
container gets whole devices. Consequences that shape the whole deployment:
| Consequence | Detail |
|---|---|
| Replica count is bounded by devices | four GPUs means at most four LocalAI pods, one device each |
| Pods are unschedulable without a device | they stay Pending, not CrashLoopBackOff |
| Node selection matters | use a node selector or taint/toleration so only GPU nodes are considered |
| Rolling updates need spare capacity | with one GPU and one replica, maxSurge: 1 deadlocks — the new pod waits for a device the old one holds |
That last row is the one that surprises people. Set maxSurge: 0 and accept downtime, or
keep a spare device.
Full manifests and the rest of the scaling story: Kubernetes and scaling.
Apple Silicon¶
The most common wasted afternoon in this ecosystem.
There is no Metal or Darwin target anywhere in LocalAI's Dockerfile or image workflows. A containerised LocalAI on Apple Silicon is CPU-only, inside a Linux VM. No flag, image tag or Docker Desktop setting changes this.
GPU acceleration on a Mac requires the native install (DMG or install script), which
pulls metal-darwin-arm64-* backends. The practical arrangement:
LocalAI native install (Metal) -> host:8080
LocalAGI container -> host.docker.internal:8080
LocalRecall container -> host.docker.internal:8080
services:
localagi:
environment:
- LOCALAGI_LLM_API_URL=http://host.docker.internal:8080
extra_hosts:
- "host.docker.internal:host-gateway"
The Apple Silicon validation was CPU-only for this reason. The CUDA 12 path on this page was validated separately on Ubuntu 24.04 amd64 with an NVIDIA Quadro RTX 6000; ROCm, Intel SYCL, Vulkan and Metal remain untested.
Proving the GPU is used¶
The image tag proves nothing. Neither does nvidia-smi showing a process.
A cpu-llama-cpp directory on a GPU image means you are running on the CPU. This is
the single most useful check on this page. Verified on a working CUDA host, /backends held
cuda12-llama-cpp and llama-cpp.
Do not judge by the process name: the running binary there was llama-cpp-cpu-all, which means
"all CPU microarchitecture variants in one binary", and nvidia-smi still attributed 3136 MiB of
VRAM to it. The directory is the signal.
Backends are downloaded as OCI artifacts at model-install time, and the variant is
chosen by hardware detection inside the container. If the device was invisible then, the
CPU backend was installed — and it persists in the /backends volume afterwards.
Restarting with the device attached does not fix it. Remove the backends volume and let it re-install:
Then:
time curl -s -o /dev/null http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"qwen3-1.7b","messages":[{"role":"user","content":"Write two sentences about vectors."}]}'
Compare warm calls only — the first includes model load either way. Reference: CPU-only on Apple Silicon returned a completion in 4 s including load.
Memory planning¶
VRAM must hold the model and the KV cache, and the cache scales with context.
| Lever | Where | Effect |
|---|---|---|
gpu_layers |
model YAML | how many layers on the device; partial offload puts the rest in system RAM |
context_size |
model YAML | KV cache reservation scales with it |
| Quantisation | the model file | Q4_K_M is the reference; smaller quantisations trade quality for memory |
Note that qwen3-1.7b's gallery config sets context_size: 8192, not the model's native
32,768. Raising it to 32,768 quadruples the KV-cache reservation — the most common cause
of an out-of-memory on load after an apparently harmless configuration change.
Loading a model may evict another, terminating that backend's process. Two models alternating on one device pay a load per request. One model per deployment is the reliable arrangement.
What a GPU does not fix¶
| Slow thing | Helped? |
|---|---|
| Token generation | yes — measured ~4.5x (30 -> 142 tok/s) |
| Embedding a large ingestion | yes per call; the call count is unchanged |
| Model load | marginally; largely I/O |
| Retrieval | no — already 30-56 ms |
| Agent wall-clock | yes, substantially — measured ~11x |
The agent row corrects an earlier, more pessimistic reading in this handbook. A GPU does not reduce the number of model calls an agent makes, so we expected only a partial gain. Measured on the same node, same model, same request, the agent went from 23.7 s to 2.12 s — a larger speed-up than the raw token-rate improvement.
Counting model calls is still the right first move when an agent is slow: a request making six calls where it should make two is a prompt or model-capability problem, and no device fixes that. But "a GPU barely helps agents" is not supported by measurement.
Count the model calls before buying hardware:
A request making six calls where it should make two is a prompt or model-capability problem, and no device fixes it.
Upstream references¶
- LocalAI container documentation — image variants and targets.
- LocalAI
Dockerfile— build targets; absence of any Metal or Darwin target. Validated against v4.8.2. - LocalAI
pkg/model/initializers.go— backend selection, load and eviction. - LocalAI
core/config/backend_config.go—gpu_layers,f16,context_size. - LocalAI
gallery/qwen3.yaml— thecontext_size: 8192the reference model actually uses. - NVIDIA Container Toolkit — Docker device passthrough.
- Kubernetes device plugins — how
nvidia.com/gpuis advertised. - Tag inventory, CPU-only latency, retrieval timings and agent model-call counts: observed 2026-08-17 on darwin/arm64.
cuda12-llama-cppin use, VRAM attribution, thecpu-allbinary naming, auto-tuning setting all layers, CUDA OOM at 17,524 and 46,297 MiB, and MoE expert offload: observed 2026-08-17 on Ubuntu 24.04 amd64, NVIDIA Quadro RTX 6000, under Docker.- GPU in Kubernetes — device plugin v0.19.3,
RuntimeClass, containerd drop-in, 2194 MiB VRAM attributed to the pod, and the 4.5x / 11x measurements: observed 2026-08-17 on k0s v1.34.3. Procedure inkubernetes/gpu/. ROCm, Intel and Vulkan untested. See version matrix.