Version matrix¶
This is our validation record, not an upstream compatibility statement. No upstream project publishes a cross-project compatibility matrix; this page exists because one is needed and none is available.
A row here means we ran it and observed the result. Nothing is marked tested that was not executed.
Versions this handbook was written against¶
| Project | Version | Released | How we read it |
|---|---|---|---|
| LocalAI | v4.8.2 | 2026-08-07 | source at tag; container reporting v4.8.2 (5ff25d9d145e0a03a5b9a3559c620f1e1204ca6d) |
| LocalAGI | v2.9.0 | 2026-05-08 | source at tag |
| LocalRecall | v0.6.4 | 2026-07-19 | source at tag |
| cogito | see below | — | source |
Validated: 2026-08-17.
What is actually inside LocalAI v4.8.2¶
The versions you get when you run LocalAI are not the standalone releases above. They are pinned commits.
| Embedded component | Pinned as | Underlying commit | Commit date |
|---|---|---|---|
| LocalAGI | v0.0.0-20260606071251-14aed1ae4336 |
14aed1ae4336 |
2026-06-06 |
| LocalRecall | v0.6.3 (release tag) |
— | 2026-06-26 |
| cogito | v0.11.1-0.20260721122412-6eece18a6bb6 |
6eece18a6bb6 |
2026-07-21 |
Two consequences worth internalising:
LocalAI does not embed LocalAGI v2.9.0. LocalAGI's Go module path has no
/v2 suffix, so Go cannot consume its v2.x tags at all. LocalAI can only pin
raw commits from main. The embedded agent platform tracks an untagged commit,
not a release.
LocalAI links a newer LocalRecall than standalone LocalAGI does. LocalAGI
v2.9.0 requires a pseudo-version built on a commit after v0.6.2; LocalAI requires
plain v0.6.3. Go's minimal version selection resolves to v0.6.3.
And the cogito split, which is the one that changes behaviour:
| Runs | cogito version | Date |
|---|---|---|
| LocalAI v4.8.2 | v0.11.1-0.20260721… |
2026-07-21 |
| LocalAGI v2.9.0 (source) | v0.9.5-0.20260315… |
2026-03-15 |
| LocalAGI v2.8.1 (the only image) | v0.9.1-0.20260216… |
2026-02-16 |
Roughly five months between the extremes. Capabilities present in the newer cogito — sub-agent spawning, KV-cache prefill, self-editing system prompts, park/resume — are not reachable from standalone LocalAGI. "Agent behaviour" therefore differs between Pattern A and Pattern B deployments of nominally the same feature.
Container images — verified to exist¶
Published registries, verified 2026-08-17:
| Project | Registry path | Note |
|---|---|---|
| LocalAI | quay.io/go-skynet/local-ai, mirror localai/localai |
|
| LocalAGI | quay.io/mudler/localagi |
see warning below |
| LocalRecall | quay.io/mudler/localrecall |
None of the three publish to ghcr.io.
LocalAI v4.8.2 tag suffixes¶
Eight, verified present:
| Tag suffix | Target |
|---|---|
| (none) | CPU, amd64 + arm64, 0.29 GB |
-gpu-nvidia-cuda-12 |
CUDA 12 |
-gpu-nvidia-cuda-13 |
CUDA 13 |
-gpu-hipblas |
AMD ROCm |
-gpu-intel |
Intel SYCL |
-gpu-vulkan |
Vulkan |
-nvidia-l4t-arm64 |
Jetson |
-nvidia-l4t-arm64-cuda-13 |
Jetson, CUDA 13 |
All-in-one (AIO) images were removed in the 4.x line. Older tags such as
latest-aio-*, -extras, -ffmpeg, -core, -cuda-11 and -intel-f16/f32
still resolve, but they are frozen builds from 2026-02-21 or earlier.
latest-cpu is stale since 2025-06-19 and is still referenced in comments in
LocalAI's own compose file. Do not use them and do not trust a tutorial that
does.
The published LocalAGI image is architecturally different from v2.9.0 source
This is the most consequential version fact in the handbook, and it is easy to miss because both are called "LocalAGI".
LocalAGI v2.8.1 — the newest published image — does not contain LocalRecall
at all. Verified: zero files import github.com/mudler/localrecall, and it
is absent from go.mod. Two capabilities that v2.9.0's source has therefore do
not exist in any runnable LocalAGI container:
| Capability | v2.9.0 source | v2.8.1 image |
|---|---|---|
| Embedded, in-process knowledge layer | yes, and it is the default | absent — retrieval is always HTTP |
Collections management API (/api/collections) |
yes, 11 routes | absent — returns Cannot GET /api/collections |
| cogito version | v0.9.5-0.20260315… |
v0.9.1-0.20260216… |
Consequences for anything you actually deploy:
LOCALAGI_LOCALRAG_URLis not an opt-in on v2.8.1; it is required for knowledge, because there is no in-process alternative to fall back to.- You cannot create or inspect collections through LocalAGI. Talk to
LocalRecall directly, or use LocalAI's
/api/agents/collections. - A separately deployed LocalRecall is mandatory, not a design choice.
Observed 2026-08-17 against the running v2.8.1 image and its source tag. Pages describing the embedded path are marked as v2.9.0-source-only; see LocalAGI ← LocalRecall.
quay.io/mudler/localagi:v2.9.0 does not exist
The highest published LocalAGI image tag is v2.8.1. The main image
build has been failing since 2026-04-15 on a self-hosted runner, while the
localagi-sshbox:v2.9.0 image did publish — which makes it easy to
conclude the main image exists when it does not. Pin v2.8.1.
Consequence: you cannot run LocalAGI v2.9.0 from a published image. The source tag exists; the container does not. Build from source or accept v2.8.1.
Reference models¶
The same two models are used throughout the beginner material, so that a failure is attributable to your configuration rather than to a model swap.
| Role | Gallery name | Size | Dims / context | Licence |
|---|---|---|---|---|
| LLM | qwen3-1.7b |
1.19 GiB (Q4_K_M) | 8,192 ctx | apache-2.0 |
| Embeddings | granite-embedding-107m-multilingual |
211 MiB (F16) | 384 dims | apache-2.0 |
The context window is 8,192, not the model's native 32,768
Qwen3-1.7B supports 32,768 tokens natively, and that figure is what most
write-ups quote. LocalAI's gallery entry does not use it: the shared qwen3
config_file sets context_size: 8192, and the qwen3-1.7b entry overrides
only parameters.model. The effective window on a stock install is therefore
8,192. Raise context_size in /models/qwen3-1.7b.yaml if you need more.
Sizes confirmed on disk: 1,282,439,296 bytes and 220,974,080 bytes respectively, observed 2026-08-17.
Why qwen3-1.7b: small enough for a CPU-only laptop, permissively licensed,
and its gallery configuration is explicitly wired for native tool calling
(use_jinja: true, use_tokenizer_template: true, function grammar disabled) —
with upstream comments citing the specific bugs that configuration avoids. Tool
calling is what makes the agent recipes work, and most small models need exactly
this treatment to do it reliably. Step up to qwen3-4b (2.33 GiB) if quality is
insufficient; the configuration is identical.
Why granite-embedding-107m-multilingual: it is upstream's own default in
three independent places — LocalAI's CLI default, LocalAGI's default, and both
projects' shipped compose files. Choosing anything else means diverging from
every default in the ecosystem for no benefit.
Total tutorial download: about 1.40 GiB of models, plus the 0.29 GB CPU image
and the llama-cpp backend, which is pulled separately on first model install.
Models to avoid¶
| Gallery name | Problem |
|---|---|
LocalAI-functioncall-llama3.2-3b-v0.5 |
The Hugging Face repository returns HTTP 401 to anonymous requests. The gallery entry is broken; the install fails at download. Verified 2026-08-17. |
bert-embeddings |
Not a BERT model. The entry resolves to Llama-3.2-1B-Instruct Q4_K_M with embeddings: true. A legacy alias that misleads. |
Our validation matrix¶
Two passes are recorded. Pass 1 exercised LocalAI alone; pass 2 brought up the full separated reference environment.
tested:
- pass: 1
date: 2026-08-17
versions:
localai: "v4.8.2 (5ff25d9d145e0a03a5b9a3559c620f1e1204ca6d)"
localagi: "not executed"
localrecall: "not executed"
environment:
architecture: arm64 (Apple Silicon)
host: macOS 26.5.1
runtime: Docker Desktop 29.5.2
gpu: none
- pass: 2
date: 2026-08-17
versions:
localai: "v4.8.2"
localagi: "v2.8.1 (image)"
localrecall: "v0.6.4 + v0.6.4-postgresql"
environment:
architecture: arm64 (Apple Silicon)
host: macOS 26.5.1
runtime: Docker Desktop 29.7.2
deployment: compose/ reference environment, Pattern B
vector_engine: postgres
gpu: none
- pass: 3
date: 2026-08-17
versions:
localai: "v4.8.2 (5ff25d9d145e0a03a5b9a3559c620f1e1204ca6d), image tag v4.8.2-gpu-nvidia-cuda-12"
localagi: "not present"
localrecall: "not present"
environment:
architecture: amd64
host: Ubuntu 24.04.3 LTS, kernel 6.8.0-88-generic
runtime: Docker 29.7.2
gpu: NVIDIA Quadro RTX 6000, 24576 MiB, driver 590.44.01
backend: cuda12-llama-cpp
note: >-
A third-party host running LocalAI for real work. Read-only inspection
plus lightweight inference against an already-resident model; we did not
install, remove or restart anything.
- pass: 4
date: 2026-08-17
versions:
localai: "v4.8.2"
localagi: "v2.8.1"
localrecall: "v0.6.4 + v0.6.4-postgresql"
environment:
platform: kubernetes
distribution: k0s v1.34.3
nodes: 4 — 1 bare-metal amd64 + 3 VMs, Ubuntu 24.04.3
storage: Longhorn (default StorageClass), ReadWriteOnce
ingress: Traefik
gpu_device_plugin: none installed
vector_engine: postgres
result: >-
All four workloads Ready and all 7 verify-stack layers pass, after fixing
three defects the Compose environment could not have revealed.
- pass: 5
date: 2026-08-17
versions:
localai: "v4.8.2-gpu-nvidia-cuda-12"
device_plugin: "nvcr.io/nvidia/k8s-device-plugin:v0.19.3"
environment:
platform: kubernetes
distribution: k0s v1.34.3
node: gpu-node — bare metal amd64, NVIDIA Quadro RTX 6000 24 GB, driver 590.44.01
container_runtime: containerd + nvidia-container-toolkit 1.18.1, RuntimeClass nvidia
backend: cuda12-llama-cpp (1.8 GiB, installed on first run)
result: >-
GPU inference in Kubernetes. 200-token completion 6.7 s CPU -> 1.4 s GPU
(~4.5x); agent with knowledge and a tool 23.7 s -> 2.12 s (~11x);
2194 MiB VRAM attributed to the pod.
- pass: 6
date: 2026-08-17
versions:
localai: "v4.8.2-gpu-nvidia-cuda-12"
localagi: "v2.8.1 (image)"
mcp_proxy: "0.12.0"
mcp_server_time: "2026.8.18"
mcp_go_sdk: "v1.2.0 (pinned by LocalAGI)"
environment:
platform: kubernetes, k0s v1.34.3
transport: SSE (Streamable HTTP attempted first, fell back)
result: >-
MCP over HTTP validated end to end with the time server. Both tools called
correctly; boundary confirmed from the LocalAGI pod IP in the server's
access log.
- pass: 7
date: 2026-08-17
versions:
localai: "v4.8.2-gpu-nvidia-cuda-12"
localagi: "v2.8.1 (image)"
environment:
platform: kubernetes, k0s v1.34.3
models: qwen3-1.7b and qwen3-4b, both resident
result: >-
Multi-agent delegation validated, same-model and cross-model. Whitelist
demonstrated narrowing the offered agent enum from 4 to 1. This completes
all nine recipes.
| # | Configuration | Deployment | Result | Notes |
|---|---|---|---|---|
| 1 | LocalAI v4.8.2, no models | Docker, CPU, darwin/arm64 | pass | Healthy in ~20 s. Agent pool auto-starts. |
| 2 | Model gallery listing | Docker, CPU, darwin/arm64 | pass | 1683 entries from index.localai.io |
| 3 | Install granite-embedding-107m-multilingual |
Docker, CPU, darwin/arm64 | pass | Also pulled the cpu-llama-cpp backend |
| 4 | Install LocalAI-functioncall-llama3.2-3b-v0.5 |
Docker, CPU, darwin/arm64 | fail | HTTP 401 from Hugging Face. Upstream gallery defect, not a local problem. |
| 5 | /v1/embeddings, single input |
Docker, CPU, darwin/arm64 | pass | 384 dims, L2-normalized, cold 3.34 s / warm 0.06–0.09 s |
| 6 | /v1/embeddings, batch of 3 |
Docker, CPU, darwin/arm64 | pass | 3 vectors, correct index ordering |
| 7 | /swagger/doc.json |
Docker, CPU, darwin/arm64 | pass | 111 paths; incomplete (/api/agents absent but live) |
| 8 | /metrics, /api/traces/summary |
Docker, CPU, darwin/arm64 | pass | See observability for what is and is not exposed |
| 9 | Gallery model install, both reference models | Compose, CPU, darwin/arm64 | fail | raw.githubusercontent.com returned HTTP 429, then 503. Every gallery config fetch failed and both installs failed — while LocalAI still logged "started and running" and /readyz returned 200 with zero models. See failure modes. |
| 10 | Backend install via /backends/apply |
Compose, CPU, darwin/arm64 | pass | llama-cpp requested; produced both llama-cpp and cpu-llama-cpp. The backend gallery is OCI-based and unaffected by the GitHub outage. |
| 11 | /models/apply with inline config_file and no url |
Compose, CPU, darwin/arm64 | fail | Get "": unsupported protocol scheme "". The documented no-URL base-config form did not work as a gallery-bypass. |
| 12 | Manual install: HF weights + hand-written model YAML | Compose, CPU, darwin/arm64 | pass | Both models resolvable in /v1/models after restart. /readyz returned 200 ~60 s after restart. |
| 13 | /v1/chat/completions, qwen3-1.7b |
Compose, CPU, darwin/arm64 | pass | 4 s including model load, max_tokens: 16 |
| 14 | Full LocalRecall round trip: create, upload, search | Compose, CPU, darwin/arm64, postgres | pass | ingest 34.9 ms, search 30.4 ms, chunk_count=1, chunk_overlap=80 honoured |
| 15 | LocalAGI agent, no tools, no knowledge | Compose, CPU, darwin/arm64 | pass | 2–3 s. usage returned as all zeros. |
| 16 | previous_response_id conversation chaining |
Compose, CPU, darwin/arm64 | pass | History carried; each reply returns a new UUID |
| 17 | Unknown previous_response_id |
Compose, CPU, darwin/arm64 | pass (surprising) | No error. Silently treated as an empty conversation. |
| 18 | Agent with a built-in tool (counter) |
Compose, CPU, darwin/arm64 | pass, with caveat | 38.7 s. Two correct tool calls; the model's narration of the result was arithmetically wrong. |
| 19 | Agent with knowledge, all three projects | Compose, CPU, darwin/arm64, postgres | pass | 2.27 s total, retrieval hop 37.19 ms. Verified by answering an invented fact. |
| 20 | LocalAGI v2.8.1 /api/collections |
Compose, CPU, darwin/arm64 | fail — absent | Cannot GET /api/collections. The route does not exist in v2.8.1. |
| 21 | scripts/verify-stack.sh, all 7 layers |
Compose, CPU, darwin/arm64 | pass | with --agent |
| 22 | Compose healthchecks | Compose, CPU, darwin/arm64 | pass | localai, postgres, localagi report healthy. localrecall cannot have one — FROM scratch, no shell. |
| 23 | qwen3-1.7b with max_tokens: 128 |
Compose, CPU, darwin/arm64 | fail — empty answer | finish_reason: length, content: ''. The reasoning consumed all 128 tokens. reasoning_content absent. |
| 24 | qwen3-1.7b with max_tokens: 600 |
Compose, CPU, darwin/arm64 | pass | finish_reason: stop, correct one-sentence answer, 371 completion tokens |
| 25 | Agent with knowledge and a tool | Compose, CPU, darwin/arm64, postgres | pass | 24.1 s; retrieved 4200, computed 42, set the counter. 2 model calls, 1 retrieval call. |
| 26 | Knowledge layer stopped mid-flight | Compose, CPU, darwin/arm64 | pass — and this is the warning | HTTP 200, status: completed, error: null, and a hallucinated "10 seconds". Logged at INFO. |
| 27 | examples/*/run.sh, seven of eight |
Compose, CPU, darwin/arm64, postgres | pass | 07-mcp not run — no MCP server available |
| 28 | mkdocs build --strict |
local, Python 3.9 | pass | 2.22 s, no warnings — after correcting an impossible version pin |
| 29 | shellcheck -S warning, all scripts |
local | pass | |
| 30 | pymdown-extensions==11.0.1 from requirements-docs.txt |
local | fail — version does not exist | Our own defect. Highest available is 10.21.3. Corrected. |
| 31 | LocalAI v4.8.2 on linux/amd64 | Docker, Ubuntu 24.04, amd64 | pass | Same build commit as pass ½. First amd64 validation. |
| 32 | CUDA 12 backend installed and used | Docker, amd64, Quadro RTX 6000 | pass | /backends holds cuda12-llama-cpp; nvidia-smi attributes 3136 MiB to the backend PID |
| 33 | Backend as an OCI artifact | Docker, amd64 | pass | metadata.json names quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-llama-cpp with a pinned digest, and alias: llama-cpp |
| 34 | Backend process and ephemeral gRPC port | Docker, amd64 | pass | lib/ld.so … llama-cpp-cpu-all --addr 127.0.0.1:45479 |
| 35 | Hardware auto-tuning | Docker, amd64, GPU | pass | effective runtime tuning … n_gpu_layers=99999999 n_batch=512 parallel=4 flash_attention=auto; disabled by LOCALAI_DISABLE_HARDWARE_DEFAULTS=true |
| 36 | Auto-tuned load of a 30B MoE on 24 GB | Docker, amd64, GPU | fail — CUDA OOM | cudaMalloc failed allocating 17,524 MiB. An 80B model asked for 46,297 MiB. Caused by n_gpu_layers=all. |
| 37 | MoE expert offload via tensor_buft_overrides |
Docker, amd64, GPU | pass | 80B MoE resident on a 24 GB card: 3136 MiB VRAM + ~46 GB RSS. Opaque options: passthrough — the string appears nowhere in LocalAI's Go source. |
| 38 | /v1/chat/completions streaming |
Docker, amd64, GPU | pass | SSE, object: chat.completion.chunk |
| 39 | Chat Completions usage |
Docker, amd64, GPU | pass | real values — {"prompt_tokens":11,"completion_tokens":1,"total_tokens":12}, unlike the agent path |
| 40 | Response id format |
Docker, amd64, GPU | pass, surprising | a bare UUID, not chatcmpl-…. Corrected an earlier claim that this was LocalAGI-specific. |
| 41 | Un-prefixed route aliases | Docker, amd64 | pass, partial | /chat/completions, /embeddings, /images/generations, /audio/transcriptions, /responses, /messages aliased; /completions, /rerank, /tokenize not. Aliasing is selective. |
| 42 | GET /system |
Docker, amd64 | pass | reports resident models and backend aliases — corrected an earlier claim that residency was unavailable |
| 43 | GET /v1/models/capabilities |
Docker, amd64 | pass | per-model capabilities and input_modalities/output_modalities |
| 44 | GET /v1/responses/{id} |
Docker, amd64 | pass | OpenAI-shaped 404 with param/type — LocalAI stores responses; standalone LocalAGI has no such route |
| 45 | /v1/messages (Anthropic shape) |
Docker, amd64 | route present | responds 400 to an empty body; the shape itself was not exercised |
| 46 | Full stack on Kubernetes | k0s v1.34.3, amd64, Longhorn, Traefik | pass | all four workloads Ready; verify-stack.sh --agent passes all 7 layers |
| 47 | PostgreSQL StatefulSet, first attempt | k0s, Longhorn RWO | fail | initdb: could not create directory … Permission denied, CrashLoopBackOff. Longhorn volume is root-owned; image runs uid 999 / gid 104. Fixed with fsGroup: 104. |
| 48 | Probes with authentication enabled | k0s | fail | LocalAI stuck 0/1 forever; LocalRecall liveness-killed in a loop. Both logged 401 from kube-probe/1.34. httpGet probes cannot read a Secret. |
| 49 | LocalRecall probe with a token | k0s | impossible | FROM scratch → no exec probe; httpGet → cannot read a Secret. Must hardcode, drop probes, or leave API_KEYS empty. |
| 50 | First inference request on a 50-pod node | k0s, amd64 | fail | FATAL -- failed to create Watcher (hpcloud/tail), pod exit 1 mid-request. Cause: fs.inotify.max_user_instances = 128 exhausted. |
| 51 | Same, after freeing 5 pods on the node | k0s, amd64 | pass | inference succeeded, zero restarts. Fragile — the durable fix is raising the sysctl. |
| 52 | Model persistence across pod recreation | k0s, Longhorn | pass | LocalAI Ready in ~20 s; models survived in the PVC |
| 53 | Retrieval across a pod boundary | k0s | pass | 55.7 ms; LocalRecall logged remote_ip 10.244.172.216, exactly the LocalAGI pod IP |
| 54 | Agent with knowledge and a tool | k0s, postgres | pass | 23.7 s; retrieved 4200, computed 42, set the counter — matching the 24.1 s Compose result |
| 55 | Pod requesting nvidia.com/gpu: 1 with no device plugin |
k0s | pass (as documented) | stays Pending: 0/4 nodes are available: 4 Insufficient nvidia.com/gpu — never CrashLoopBackOff |
| 56 | nginx ingress annotations on Traefik | k0s, Traefik | silently ignored | Traefik has no per-Ingress read-timeout annotation; needs static config or a ServersTransport CRD |
| 57 | Device plugin on a stock k0s worker | k0s v1.34.3 | fail | no runtime for "nvidia" is configured. /etc/k0s/containerd.d/ is empty by default; Docker's toolkit config does not touch k0s's containerd. |
| 58 | containerd drop-in with SystemdCgroup = true |
k0s | fail | runtime lookup succeeded, then runc: expected cgroupsPath to be of format "slice:prefix:name" … got "/kubepods/besteffort/…". k0s uses cgroupfs — must be false. |
| 59 | Device plugin v0.19.3 after both fixes | k0s, gpu-node | pass | 1/1 Running; nvidia.com/gpu capacity=1 allocatable=1; logs Registered device plugin for 'nvidia.com/gpu' with Kubelet |
| 60 | First GPU deployment with liveness probe only | k0s | fail | CUDA backend is 1.8 GiB and downloads before the HTTP listener starts, so /readyz refuses connections and liveness killed it at ~120 s — 5 restarts, download restarting from zero each time. |
| 61 | Same, with a startupProbe added |
k0s | pass | Ready in ~3 min, zero restarts. The CPU backend fits inside the liveness window, so this defect appears only on first GPU deployment. |
| 62 | CUDA backend auto-selected in Kubernetes | k0s, gpu-node | pass | /backends → cuda12-llama-cpp; log capability="nvidia-cuda-12" |
| 63 | VRAM attributed to the pod's backend process | k0s, gpu-node | pass | 2194 MiB against /backends/cuda12-llama-cpp/lib/ld.so; reads 0 MiB until the first request (lazy backend start) |
| 64 | GPU vs CPU, 200-token completion, same node/model/prompt | k0s, gpu-node | pass | CPU 6.67 / 6.78 s (~30 tok/s) → GPU 1.41 / 1.50 s (~142 tok/s) = ~4.5x |
| 65 | GPU vs CPU, agent with knowledge and a tool | k0s, gpu-node | pass | 23.7 s → 2.12 s ≈ 11x. Corrected an earlier claim that a GPU helps agent latency "only partly". |
| 66 | All 7 verify-stack layers on GPU | k0s, gpu-node | pass | agent request 4 s |
| 67 | LocalAGI image runtimes for stdio MCP | k0s | fail — none present | node, npx, python3, uvx all MISSING in v2.8.1. The npx/uvx reference-server ecosystem cannot run as a stdio child as shipped. |
| 68 | ghcr.io/sparfenyuk/mcp-proxy:v0.12.0 wrapping uvx mcp-server-time |
k0s | fail | FileNotFoundError: [Errno 2] No such file or directory: 'uvx' — that image does not bundle uv either |
| 69 | mcp-proxy + mcp-server-time on a python:3.13-slim base |
k0s | pass | serves SSE at /sse; resolved mcp-proxy 0.12.0, mcp-server-time 2026.8.18 |
| 70 | MCP discovery with the server not yet serving | k0s | fail, silently | Failed to connect to MCP server via SSEClientTransport … connection refused, then Done populating actions and Agent started. Agent runs with no MCP tools; not retried later. |
| 71 | MCP discovery after the server is up | k0s | pass | no connect error; tools populated |
| 72 | get_current_time via MCP |
k0s, GPU | pass | {"timezone":"Asia/Tokyo"} → 2026-08-22T08:26:14+09:00, Saturday, is_dst false. 2.6 s |
| 73 | convert_time via MCP |
k0s, GPU | pass | Europe/Rome 09:00 → America/New_York 03:00, −6.0h — correct for August |
| 74 | MCP boundary crossing confirmed | k0s | pass | server logged POST /messages/?session_id=… from 10.244.172.216, exactly the LocalAGI pod IP |
| 75 | Streamable HTTP → SSE fallback | k0s | pass (as documented) | mcp.go:158-166; the /sse endpoint took the SSE path |
| 76 | call_agents tool schema |
k0s | pass | action key is call_agents, tool name is call_agent; agent_name is an enum, so an agent name cannot be hallucinated |
| 77 | call_agents with no whitelist |
k0s | pass — and this is the risk | enum listed all 4 pool agents, including the coordinator itself |
| 78 | whitelist filtering |
k0s | pass | 4 agents → ["unit-converter"] |
| 79 | blacklist filtering |
k0s | pass | blacklist=k8s-probe,mcp-probe → the other two remained |
| 80 | Same-model delegation | k0s, GPU | pass | 12 km → 7.456 miles. 10.6 s cold, 7.5 s warm; the specialist's own loop was ~9 s of it |
| 81 | Specialist History after delegation |
k0s | empty — not a failure | History records action results; a tool-less specialist has none. The log is the proof it ran |
| 82 | Delegated message content | k0s | rewritten | the coordinator composed "Calculate 47 requests/second * (3*3600 + 25*60) seconds." rather than forwarding the user's prose |
| 83 | Cross-model delegation | k0s, GPU | pass | log confirmed agent=router model=qwen3-1.7b → agent=deep-thinker model=qwen3-4b; 37.5 s cold, 7.8 s warm |
| 84 | Whether the cross-model split improved the answer | k0s, GPU | no difference on this test | qwen3-1.7b answered 578,100 correctly unaided, so the routing mechanism was validated but no quality gap was demonstrated |
| 85 | Three models resident simultaneously | k0s, GPU | pass | qwen3-1.7b 2202 MiB + granite 234 MiB + qwen3-4b 4850 MiB = 7294 MiB, no eviction |
| 86 | Shipped 07-ingress.yaml as written |
k0s, Traefik | unusable | host was the placeholder agents.example.com and every annotation was nginx.ingress.kubernetes.io/*; the object still reported an address |
| 87 | Ingress via *.<ip>.sslip.io host |
k0s, Traefik | pass | resolves through public DNS with no zone edit; makes an ingress testable the moment it is applied |
| 88 | Dual-host Ingress via a YAML anchor | k0s, Traefik | pass | one backend definition serving both <svc>.example.com and the sslip.io host |
| 89 | LocalAI through the ingress | k0s, Traefik | pass | /readyz 200, /v1/models → 3 models, /system → 3 backends loaded |
| 90 | LocalAGI through the ingress | k0s, Traefik | pass | /api/agents → 6 agents; /v1/responses answered |
| 91 | LocalRecall through the ingress | k0s, Traefik | pass | /api/collections → ["k8s-probe"] |
| 92 | verify-stack.sh with all URLs on the ingress |
k0s, Traefik, GPU | pass — all 7 layers | first run with no port-forward; included the full ingest → embed → search round trip and a 2 s agent request |
| 93 | Longest request achievable, through the ingress | k0s, Traefik, GPU | 36 s | 4000-token essay, finish_reason: stop. 7000-token forced output was 24 s (~290 tok/s); a delegated cross-model request 18 s |
| 94 | Whether Traefik cuts off a slow agent request | k0s, Traefik | it does not | writeTimeout — the setting that governs a slow response — defaults to 0, unlimited. readTimeout 60 s covers reading the request only. Documented, not tested past 36 s |
| 95 | nginx.ingress.kubernetes.io/* annotations on Traefik |
k0s, Traefik | silently ignored | no warning, no event; an ignored annotation is indistinguishable from a working one |
| 96 | basicAuth Traefik Middleware manifest |
k0s, Traefik | schema valid, not enabled | middlewares.traefik.io CRD present, --dry-run=client passed; never put in front of a live route |
| 97 | LocalAI agent pool enabled (LOCALAI_DISABLE_AGENTS=false) |
k0s, GPU | pass | /api/agents 200 where it was 404; actions: 40, connectors: 9 |
| 98 | Agent-pool route prefix | k0s | /api/agents/collections |
not /api/collections — collides with neither LocalRecall's nor LocalAGI's API |
| 99 | Agent-pool routes in /swagger/doc.json |
k0s | absent | swagger documents only /api/agent/tasks and /api/agent/jobs; pool CRUD is undocumented |
| 100 | GET /api/agents response shape |
k0s | summary, not a list | {"agentCount":1,"agents":[...],"actions":40,"connectors":9} — LocalAGI returns an array |
| 101 | Collection create / upload / search in LocalAI | k0s, GPU | pass | create 201; multipart upload 200; search returned similarity 0.693 |
| 102 | LocalAI vs LocalRecall response envelope | k0s | not wire-compatible | {"collections":[..],"count":1} vs {"success":true,"data":{...}} |
| 103 | DELETE a LocalAI collection |
k0s | query param only | ?name=X → 200; /collections/X → 404 |
| 104 | How an agent selects its collection | k0s | by agent name | no collection field in the 57-field config; agent handbook reads collection handbook |
| 105 | Agent answering from LocalAI's own knowledge | k0s, GPU | pass | correct retrieval of a synthetic sentinel, 53 s cold including embedding-backend load |
| 106 | local_rag_url on a LocalAI agent |
k0s, GPU | accepted and SILENTLY IGNORED | value persists; LocalAI logs Chromem collection ... dbPath="/data/collections" anyway; target LocalRecall logged zero requests; seeding LocalAI's own collection fixed the answer |
| 107 | kb_results > document count on chromem |
k0s | hard error, logged at INFO | nResults must be <= the number of documents in the collection → No similar strings found in KB → a confident "I do not know" with nothing at ERROR |
| 108 | /data unmounted in the base manifest |
k0s | defect found and fixed | --data-path holds collectiondb, agent state, tasks, jobs; it was the container's writable layer, so ingestion was discarded on every rollout |
| 109 | Strategic-merge patch over a configMapKeyRef |
k0s | rejected without an explicit null | env[0].valueFrom: Invalid value: "": may not be specified when 'value' is not empty; valueFrom: null in the same patch fixes it |
| 110 | POST /api/agents/import |
k0s | pass — 201 | body is the GET /api/agents/{name}/export object; empty body returns 400, which proves the route exists |
| 111 | Imported agent's knowledge | k0s, GPU | empty — by design, and it looks broken | collections bind by agent name, so an import under a new name auto-creates an empty collection and then denies all knowledge via the nResults path |
| 112 | Import control in the WebUI | k0s | present, not feature-gated | a <label class="btn btn-secondary"> wrapping a hidden <input type=file> on /app/agents; only the Agent Hub link is conditional |
| 113 | Import control in the zero-agents empty state | k0s | UI defect | agents-import-input is on the <input>, but the rule is .agents-import-input input[type=file]{display:none} — needs the class on an ancestor, so the raw native file picker renders |
| 114 | Deep-linking to import mode | k0s | not possible | "Import Agent" is gated on router state importedConfig; /app/agents/new shows "Create Agent" |
| 115 | DELETE /api/agents/collections?name=X |
k0s | silent no-op — but it is the wrong route | returns 200 {"status":"ok"} and changes nothing. Deletion is implemented elsewhere: see row 123 |
| 116 | How an agent binds to a collection | k0s, GPU | agent name, lowercased | two canaries: agent AutoResearchAgent retrieved from collection autoresearchagent, not from AutoResearchAgent. Both exist as separate dirs under /data/assets/ |
| 117 | Pointing an agent at an arbitrary collection | k0s | not possible | no collection field in the 57-field config; the WebUI agent form contains no collection control at all |
| 118 | kb_results default vs a small collection |
k0s, GPU | retrieval silently dead | a real agent had kb_results: 10 against a collection whose chunk count was lower → nResults error → every answer a confident denial. Reduced, retrieval worked on the next request. The unit is chunks — see row 143 |
| 119 | can_stop_itself: true |
k0s, GPU | answers become 500s | {"error":{"message":"interrupted via ToolCallCallback","type":"server_error"}}; false answered the identical prompt |
| 120 | long_term_memory + summary_long_term_memory past context |
k0s, GPU | SIGSEGV — whole process dies | summarization fails with 14899 tokens exceeds the available context size (8192), then nil-pointer panic in saveCurrentConversation, knowledgebase.go:147. Exit 2, pod restart, all models unloaded. Reached after 4 messages on a 4B model |
| 121 | State survival across that crash | k0s | pass — because of the PVC | 3 agents and 6 collections intact after the restart; without the localai-data PVC all of it would have been lost |
| 122 | Ingesting GitHub blob URLs |
k0s | stores page chrome, not the doc | chunks were Breadcrumbs, Copy raw file, Blame; at MAX_CHUNKING_SIZE=400 boilerplate fills whole chunks. Use raw.githubusercontent.com |
| 123 | DELETE /api/agents/collections/{c}/entry/delete |
k0s | pass — this is the working delete | body {"entry":"<uuid>/<file>"}; returns {"count":N,"remaining_entries":[...]}. Also present: POST .../reset and POST .../sources. None are in swagger; found in the WebUI's API bundle |
| 124 | summary_long_term_memory: false as the crash mitigation |
k0s, GPU | pass | identical Saving conversation ... conversation size=4 step, no summarization request and no panic. Removes the code path rather than postponing it |
| 125 | enable_planning on a 4B model with RAG context |
k0s, GPU | fails the request | failed to execute planning: failed to execute plan: no subtasks found in plan; two consecutive turns |
| 126 | Plain RAG after disabling 10 advanced flags | k0s, GPU | pass — 4.9 s | same agent that had failed three different ways answered correctly twice in a row, quoting two facts from a sentinel document. Retrieval confirmed in the log |
| 127 | long_term_memory write-back target |
k0s, GPU | the agent's own retrieval collection | conversation files land beside your documents, compete for kb_results slots and change the count kb_results must stay under |
| 128 | Qwen3-Next-80B-A3B Q4_K_M with tensor_buft_overrides:exps=CPU in Kubernetes |
k0s, GPU | pass | 3168 MiB VRAM of 24576 (14% of the card), 25.0 tok/s at 600 completion tokens, 100 s cold first request |
| 129 | Base limits.memory: 8Gi against a 45 GiB model |
k0s, GPU | the Kubernetes-specific blocker | absent under Docker, which has no limit by default; appears in no model YAML |
| 130 | What the cgroup actually charges under mmap: true |
k0s, GPU | deferred and prompt-dependent | memory.current 7.9 GiB after one prompt, 8.0 after two, 8.5 after five and still climbing; kubectl top showed 6.7 GiB for a 45 GiB model. Sizing the limit from observation is therefore wrong — size it from the file |
| 131 | Longhorn for a 45 GiB model volume | k0s | wrong store | numberOfReplicas: 3 → a 120 GiB claim costs 360 GiB cluster-wide, and every download write replicates twice over the network. Non-GPU nodes had 87-104 GiB free |
| 132 | local-path for the model volume |
k0s | pass | WaitForFirstConsumer provisions on the GPU node; claim sits Pending until a pod claims it, which is correct. Trade-off: the volume dies with the node |
| 133 | 45 GiB fetch as a Job vs /models/apply |
k0s | Job is better | resumable with curl -C -, watchable, and LocalAI keeps serving throughout; sha256 verified 83481c75…c81367 |
| 134 | curlimages/curl writing to a fresh local-path volume |
k0s | fails | uid 100 vs a root-owned volume: mkdir: can't create directory ... Permission denied before any transfer. Needs runAsUser: 0 |
| 135 | Two pods mounting one ReadWriteOnce claim |
k0s | legal on the same node | the fetch Job and the LocalAI pod shared the claim; the nodeSelector is what guarantees co-location |
| 136 | kubectl exec pod -- sh -c 'cat > f' < local without -i |
k0s | writes a 0-byte file, exits 0 | check the byte count, not the exit status |
| 137 | A Thinking model with a small max_tokens |
k0s, GPU | empty content |
max_tokens: 80 was consumed entirely by the reasoning block; reads as a load failure and is not one |
| 138 | Viewing a stored document through the ingress | k0s, Traefik | 400 — a regression from publishing | both UIs encodeURIComponent a <uuid>/<filename> key, so the / becomes %2F; Traefik rejects it. Surfaces as Failed to load entry content: HTTP 400 |
| 139 | Is the %2F rejection specific to collections? |
k0s, Traefik | no — blanket | /readyz%2Ffoo also returns 400, so any encoded slash on any route fails |
| 140 | Same request straight to the pod | k0s | 200 | LocalAI decodes %2F correctly and logs the decoded path; the literal-slash form also works through Traefik |
| 141 | Is it a permissions problem? | k0s | no | LocalAI runs as uid=0(root); /data/assets and the files beneath are root:root and mode-readable |
| 142 | Can a Traefik Middleware fix it? |
k0s, Traefik | no | path parsing precedes middleware, so the 400 is returned before any route middleware runs |
| 143 | What nResults actually counts |
k0s | chunks, not files | a collection with one file that chunked into 11 accepted max_results up to 11 and failed at 12. The error says "documents", meaning chromem documents. /entries count is a lower bound only |
| 144 | LoadBalancer Service bypassing Traefik |
k0s, MetalLB | pass — fixes the 400 | identical %2F request: 200 via the LoadBalancer, 400 via the Ingress. Full surface reachable; no shared Traefik config touched |
| 145 | Retrieval depth after correcting kb_results |
k0s, GPU | qualitative jump | at kb_results: 1 the agent named the file it could not summarise; at 4 it answered with the document's actual content in 28 s |
A reproduced failure worth knowing¶
Row 9 is the most useful failure in this table, because it will happen to readers and it does not look like a failure.
When raw.githubusercontent.com rate-limits you — HTTP 429 — LocalAI's model
gallery cannot resolve any entry, so model installation fails at startup.
LocalAI nonetheless logs:
and /readyz returns 200 with zero models installed. One of the error lines is
itself defective:
An error log with a nil error. Practical lessons: /readyz is not a readiness
signal for anything except the listener, and the only reliable check is
GET /v1/models. This is precisely why
verify-stack.sh
treats "models resolvable" as a distinct layer from "process reachable".
The backend gallery was unaffected, because backends are OCI artifacts from a container registry rather than YAML on GitHub.
Not yet validated¶
Recorded honestly, because the gaps matter:
| Configuration | Status |
|---|---|
| LocalAGI v2.9.0 in any form | not executed — no published image; not built from source |
| Embedded (in-process) knowledge layer | tested on LocalAI v4.8.2 (rows 97-107). Still not executed on LocalAGI, where it is absent from v2.8.1 |
LocalAGI's /api/collections API |
not executed — absent from v2.8.1 |
Standalone LocalRecall with chromem engine |
not executed — only postgres was exercised |
| Hybrid search weight tuning, BM25 behaviour | not executed — the engine was used, the weights were not varied |
| Long-term memory write-back | not executed — observed disabled in the log |
| ROCm, Intel SYCL, Vulkan, Metal | not executed — only CUDA 12 was available |
| GPU for LocalAGI or LocalRecall | n/a — neither ever uses a device |
Distributed llama.cpp (LLAMACPP_GRPC_SERVERS) |
not executed — the binary is present in the bundle |
/v1/messages, /v1/rerank, and the face/voice/vision endpoints |
not exercised — routes confirmed present only |
| Distributed mode (NATS + PostgreSQL) | not executed |
| Pattern A with agents enabled and knowledge | tested — rows 97-107, kubernetes/pattern-a/ |
LocalAI's agent pool on the postgres vector engine |
not executed — only chromem was exercised |
| One PostgreSQL database shared by LocalAI's pool and a standalone LocalRecall | not executed — risks a schema migration running under the other process |
| TLS on the ingress | not executed — HTTP only; no certificate issuer in the cluster |
| The basic-auth middleware actually in front of a route | not executed — manifest validated, never enabled |
| A request longer than 60 s through an ingress | not achievable here — the GPU answers too fast; 36 s was the maximum |
Every page describing the above is source-verified, not tested. The distinction is maintained in the prose.
Platform notes¶
Docker on macOS has no Metal access. Verified: there is no Metal or Darwin
target anywhere in LocalAI's Dockerfile or image workflows. A containerised
LocalAI on Apple Silicon runs CPU-only inside a Linux VM. GPU acceleration on a
Mac requires the native install (DMG or install script), which pulls
metal-darwin-arm64-* backends.
The practical arrangement on a Mac: run LocalAI natively, run LocalAGI and
LocalRecall in containers pointed at host.docker.internal:8080.
No Intel-Mac binary is published. Release assets cover
linux-amd64, linux-arm64, darwin-arm64 and a DMG.
Homebrew is not a distribution channel for LocalAI. Every brew reference in
upstream material is a build dependency, not an install path.
Keeping this page honest¶
When you validate a configuration, add a row with the date and the observed version strings, and update the "not yet validated" table. When you cannot validate something, leave it in the second table. An empty result is information; a fabricated one destroys the point of the handbook.
See Contributing for the evidence rules.
Upstream references¶
- LocalAI releases — v4.8.2, 2026-08-07.
- LocalAI
go.mod— pinned LocalAGI, LocalRecall and cogito versions. - LocalAGI releases — v2.9.0, 2026-05-08.
- LocalRecall releases — v0.6.4, 2026-07-19.
- LocalAI
gallery/qwen3.yaml—qwen3-1.7btool-calling configuration. - LocalAI
core/cli/run.go— default embedding model. - Traefik responding timeouts —
readTimeout60 s,writeTimeout0,idleTimeout180 s. Read 2026-08-22. - Image tags, model sizes and failure results: observed 2026-08-17.
- Ingress results: observed 2026-08-22, Traefik v3.6.6, k0s v1.34.3.