LocalAI overview¶
LocalAI is a single Go server process. It terminates HTTP, decides which model a request means, and executes nothing itself: inference runs in separate backend processes that LocalAI starts, supervises and speaks gRPC to.
flowchart LR
APP["client"]
subgraph P1["local-ai process"]
API["HTTP API<br/>Echo v4"]
ML["model loader<br/>+ watchdog"]
end
subgraph P2["backend process<br/>e.g. cpu-llama-cpp"]
ENG["inference engine"]
end
W[("weights + YAML")]
APP -->|"HTTP /v1/chat/completions"| API
API -.->|in-process| ML
ML -->|"gRPC 127.0.0.1:<freeport>"| ENG
ENG -.->|"fs: /models"| W
The gRPC hop is real TCP on loopback, not a Unix socket and not an in-process
call. pkg/model/initializers.go allocates a free port and formats
127.0.0.1:<port> for every locally spawned backend; there is no
net.Listen("unix" anywhere in pkg/, core/ or backend/
(source-verified, v4.8.2). The one in-process transport that exists,
pkg/grpc/embed.go, is selected only by test code.
Small core, not a bundle¶
A stock container ships zero models and zero backends. Both are downloaded on demand: models from a model gallery, backends as OCI artifacts from a container registry.
Tested against localai/localai:latest (digest id df2919064853, arm64,
reporting v4.8.2 (5ff25d9d145e0a03a5b9a3559c620f1e1204ca6d)), observed
2026-08-17:
| Probe | Response |
|---|---|
GET /v1/models |
{"object":"list","data":[]} |
GET /backends |
[] |
GET /system |
{"backends":[],"loaded_models":[]} |
GET /models/galleries |
one gallery, https://index.localai.io/models |
The CPU image is 0.29 GB compressed (Docker Hub, measured 2026-08-17); the GPU
variants run 1.11–6.13 GB. That is the floor. A first useful run costs image +
backend artifact + model weights. Installing
granite-embedding-107m-multilingual also pulled
quay.io/go-skynet/local-ai-backends:latest-cpu-llama-cpp (42 MiB) without being
asked, because the model's gallery entry declares a backend and backend gallery
autoload is on by default (tested 2026-08-17).
This is deliberate. The all-in-one images that bundled a preset of models were
dropped in the 4.x line; upstream's own wording is that maintaining them across
every hardware variant "stopped being worth what they gave people"
(documented, website/content/blog/what-landed-in-localai-4-0.md).
Two binaries, one server¶
| Binary | Source | Role |
|---|---|---|
local-ai |
cmd/local-ai/main.go |
The server and every CLI subcommand |
LocalAI.app launcher |
cmd/launcher/main.go |
A Fyne/systray desktop wrapper. Not the server; it supervises one |
internal/version.go sets Version and Commit as link-time variables, so a
plain go build reports an empty version string. A binary that answers
GET /version with nothing was built from source, not installed from a release
(source-verified, v4.8.2).
CLI surface¶
core/cli/cli.go defines the complete top-level command set. run is the
default command, so local-ai <model-name> is local-ai run <model-name>.
| Subcommand | Purpose | Starts an HTTP server? |
|---|---|---|
run (default) |
The LocalAI server | Yes, :8080 |
chat |
Terminal agent against a LocalAI server | No |
federated |
P2P federated load balancer | Yes, :8080 |
models |
list, install |
No |
backends |
list, install, uninstall, upgrade |
No |
tts |
One-shot text to speech | No |
sound-generation |
Audio from text/audio | No |
transcript |
Audio to text | No |
p2p-worker |
P2P workload distribution (p2p-llama-cpp-rpc, p2p-mlx, llama-cpp-rpc, mlx-distributed, vllm, ds4-distributed) |
No |
worker |
Distributed-mode worker (NATS + registration) | Yes, gRPC base − 1 |
agent-worker |
Executes agent chats from a NATS queue | No |
util |
gguf-info, create-oci-image, hf-scan, usecase-heuristic |
No |
agent |
run, list — agents without the server |
No |
mcp-server |
LocalAI's admin surface as a stdio MCP server | No |
explorer |
P2P network directory | Yes, :8080 |
completion |
bash/zsh/fish completion scripts | No |
Four global flags apply to every subcommand (core/cli/context/context.go):
--debug (deprecated), --log-level, --log-format, --log-dedup-logs.
Two renames worth knowing before you copy a command from anywhere:
workermeant the P2P worker before v4.1.0 and means the NATS distributed worker now. The P2P one isp2p-worker. LocalAI's own printed join instructions still say the old form (core/cli/run.goprintslocal-ai worker p2p-llama-cpp-rpc), as do several upstream docs pages. The correct command at v4.8.2 islocal-ai p2p-worker p2p-llama-cpp-rpc(source-verified, v4.8.2).local-ai agent runstarts no listener at all. It needs a reachable OpenAI-compatible endpoint (--api-url), but serves no API of its own.
What local-ai run starts, in order¶
RunCMD.Run in core/cli/run.go (source-verified, v4.8.2):
| # | Step | Consequence for an operator |
|---|---|---|
| 1 | Warn on deprecated flags | — |
| 2 | Collect systemd socket-activation listeners | The only way /readyz answers during preload |
| 3 | MkdirAll on backends and models paths, mode 0750 |
— |
| 4 | Build the system state (paths, capability, tags) | Capability decides which backend build gets pulled |
| 5 | Assemble ~200 application options from flags/env | — |
| 6 | If --preload-backend-only: construct, then exit |
No HTTP server at all |
| 7 | application.New(opts...) — blocking, does all preload |
Tens of GB of downloads can happen here |
| 8 | Public-bind safety check | Refuses to start on a public address with no auth unless --allow-insecure-public-bind |
| 9 | http.API(app) builds the Echo server |
Routes and middleware are registered here, nothing is listening yet |
| 10 | Start P2P if a token exists | — |
| 11 | Register the graceful-termination handler | Stops backends on SIGINT/SIGTERM |
| 12 | Spawn a goroutine that waits for the listener to accept, then StartAgentPool() |
Agents come up after the API is answering |
| 13 | appHTTP.Start(listenAddress) — blocks |
The socket binds here, last |
Step 13 being last is the single most consequential ordering fact in the whole
system. During step 7 the port is not open, so probes get connection refused
rather than a 503. Step 12 exists because agent knowledge-base backends call the
embeddings API on this same process over loopback HTTP — starting the pool before
the listener accepts would deadlock. See
architecture for both in detail.
Where state lives¶
Defaults are relative to kong's ${basepath}, which is the process working
directory, not a fixed system path. In the container WORKDIR / makes them
absolute (Dockerfile, source-verified v4.8.2).
| Path (container) | Flag / env | Holds |
|---|---|---|
/models |
--models-path / LOCALAI_MODELS_PATH |
Weights and per-model YAML |
/backends |
--backends-path / LOCALAI_BACKENDS_PATH |
Installed backend artifacts |
/configuration |
--localai-config-dir / LOCALAI_CONFIG_DIR |
api_keys.json, external_backends.json, runtime_settings.json |
/data |
--data-path / LOCALAI_DATA_PATH |
Agent state, jobs, auth DB, HMAC secret, traces, MITM CA |
/var/lib/local-ai/backends |
LOCALAI_BACKENDS_SYSTEM_PATH |
Read-only, package-installed backends |
Running local-ai run from a different directory silently uses a different
models/backends/data tree. Set the three path variables explicitly outside
containers.
Reading order¶
| You want | Page |
|---|---|
| Process model, middleware, startup ordering, readiness | architecture |
Images, binaries, volumes, docker run lines |
installation |
| Env vars and flags, grouped | configuration |
| Model YAML, gallery, install jobs | models |
| Backend artifacts, gRPC contract, eviction | backends |
| Every route, and which dialect it speaks | api |
/v1/embeddings, with measured numbers |
embeddings |
| Accelerators and what actually works | gpu |
| Diagnosing a specific failure | troubleshooting |
Upstream references¶
cmd/local-ai/main.go— entrypoint, dotenv load order, kong parsing. Validated against v4.8.2.core/cli/cli.go— the complete subcommand set. Validated against v4.8.2.core/cli/run.go—runstartup ordering, defaults, safety checks. Validated against v4.8.2.pkg/model/initializers.go— backend spawn,127.0.0.1:<freeport>addressing. Validated against v4.8.2.internal/version.go— link-time version variables. Validated against v4.8.2.Dockerfile—WORKDIR /,VOLUME /models /backends /configuration /data. Validated against v4.8.2.- LocalAI release v4.8.2 — validated 2026-08-17.
- Empty-instance probes, image digest, backend auto-install: observed 2026-08-17 on
localai/localai:latestreportingv4.8.2 (5ff25d9d).