Skip to content

LocalAI ← LocalRecall

The knowledge layer reaching the model runtime. LocalRecall computes no embeddings itself, so this edge is not optional: without it, a collection cannot be written to or queried at all.

There are actually two distinct edges between these projects, and conflating them causes real confusion:

Edge What LocalAI provides When it exists
Embeddings POST /v1/embeddings always
Vector storage POST /stores/* only when VECTOR_ENGINE=localai

The first is the architecture. The second is one of three storage options, and the least complete of them.

Edge 1 — embeddings

flowchart LR
  subgraph LR1["localrecall process"]
    ING["ingest + chunk"]
    ENG["vector engine"]
    CL["go-openai client"]
  end
  subgraph LAI["local-ai process"]
    EMB["/v1/embeddings"]
  end
  BE["embedding backend"]
  VS[("vector store")]
  ING -.->|in-process| ENG
  ENG -.->|in-process| CL
  CL -->|"HTTP POST, network"| EMB
  EMB -->|gRPC| BE
  ENG -->|"file or SQL"| VS

As with LocalAGI's edge, this is a plain github.com/sashabaranov/go-openai client. LocalRecall does not know or care that the endpoint is LocalAI.

Configuration

Variable Required Default Meaning
OPENAI_BASE_URL effectively yes (none — and this breaks) base URL of the embeddings server
EMBEDDING_MODEL yes (none — and this breaks) embedding model name
OPENAI_API_KEY no empty bearer token

Neither required variable is validated at startup. Both failure modes are quiet and both are worth recognising:

OPENAI_BASE_URL unset. main.go assigns config.BaseURL = openAIBaseURL unconditionally, after openai.DefaultConfig has already set the real OpenAI URL. An empty value therefore does not fall back to a default — it overwrites the default with nothing, and every embedding request goes to a bare relative /embeddings. The process starts happily; the first ingestion fails.

EMBEDDING_MODEL unset. There is no default. An empty model name goes on the wire and LocalAI rejects it.

The full treatment, including which call sites are affected, is in LocalRecall embeddings.

The /v1 question, answered differently here

Unlike cogito, LocalRecall's client is the standard go-openai client, which appends the OpenAI path itself. Set the base URL without /v1:

Target OPENAI_BASE_URL
LocalAI http://localai:8080
Any OpenAI-compatible server http://host:port

So the two edges in this stack want the base URL written differently, for different reasons. Against LocalAI both happen to work, which is exactly why the inconsistency stays hidden until you substitute another server. Keep the api-flow /v1 trap in mind whenever you change either one.

The dimension coupling

This is the most consequential property of the edge, and it is not enforced anywhere.

A vector's dimension is a property of the embedding model. Every vector in a collection must come from the same model, because similarity between vectors from different models is meaningless even when the dimensions happen to match.

Measured on the reference model used throughout this handbook (granite-embedding-107m-multilingual): 384 dimensions, L2-normalized, so cosine similarity is a plain dot product.

What happens if you change EMBEDDING_MODEL against an existing collection depends on the engine, and neither outcome is good:

Situation Result
New model, different dimension writes fail on dimension mismatch
New model, same dimension writes succeed and retrieval silently degrades

The second is the dangerous one: no error, no warning, just worse answers. Treat EMBEDDING_MODEL as part of the collection's identity. Changing it means re-ingesting, not migrating — see changing the embedding model.

Cost per operation

Each edge crossing is one HTTP request plus one gRPC call inside LocalAI:

Operation Embedding calls
Ingest a document one per chunk
Search a collection exactly one, for the query

Ingestion is therefore the expensive side, and it is the side people benchmark last. A 400-character chunk size against a moderately sized manual is thousands of embedding calls. Whether they are batched depends on the engine — see batching differs by engine.

Observed for the reference model: cold call 3.34 s including model load, warm calls 0.06–0.09 s. Model load happens once; plan ingestion around the warm figure and the count of chunks.

Edge 2 — LocalAI as the vector store

VECTOR_ENGINE=localai selects a third mode in which LocalAI holds the vectors too, through its /stores/* API. LocalRecall talks to it with a small hand-written HTTP client whose first line calls itself a duplicate of LocalAI's own store client.

In this configuration LocalAI is both the embedder and the database:

flowchart LR
  subgraph LR1["localrecall process"]
    ENG["localai engine"]
  end
  subgraph LAI["local-ai process"]
    EMB["/v1/embeddings"]
    ST["/stores/set<br/>/stores/find"]
  end
  ENG -->|"HTTP"| EMB
  ENG -->|"HTTP"| ST

Attractive on paper — one fewer stateful component — but several methods on this engine return not implemented, including Reset, Count and GetEmbeddingDimensions. Operations the other engines support therefore fail here.

Do not choose it for new work. chromem for a single node, PostgreSQL when you need hybrid search or shared access. The comparison table is in storage.

Why PostgreSQL earns its place

Worth stating here because it is the one dependency this handbook's reference Compose environment adds beyond the three projects.

Hybrid search — combining vector similarity with BM25 lexical scoring — is implemented only in the PostgreSQL engine, and requires the pg_textsearch extension for BM25 indexing. Weights are tunable:

Variable Default
HYBRID_SEARCH_VECTOR_WEIGHT 0.5
HYBRID_SEARCH_BM25_WEIGHT 0.5
BM25_TEXT_CONFIG english

So the choice is not "PostgreSQL because production needs a database". It is: pure vector search on a file, or vector plus lexical search on PostgreSQL. Queries containing exact identifiers — error codes, function names, product SKUs — are where the difference shows, because embeddings are poor at exact-token matching and BM25 is good at it.

Verifying this edge in isolation

Prove embeddings work before blaming retrieval.

curl -s http://<localai-host>:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"granite-embedding-107m-multilingual","input":"test"}' \
  | jq '.data[0].embedding | length'

Expected: 384. If this returns an error, no collection will work and the problem is not in LocalRecall.

Then from inside the LocalRecall container, which is where its DNS applies:

docker exec localrecall wget -qO- \
  --header='Content-Type: application/json' \
  --post-data='{"model":"granite-embedding-107m-multilingual","input":"test"}' \
  http://<localai-host>:8080/v1/embeddings

Then confirm the collection layer, which exercises the edge end to end:

curl -s -X POST http://localhost:8081/api/collections \
  -H 'Content-Type: application/json' -d '{"name":"probe"}'

A 502 with "Vector backend unavailable" here means collection creation reached the embedding call and it failed — the edge is broken, not the API. That distinction is deliberate in the code: an earlier version called os.Exit on this path and crash-looped the server during transient embedding outages.

There is no health endpoint on LocalRecall. GET /api/collections is the closest thing, and it answers from disk without touching the embedding server, so a 200 from it proves the process is up and proves nothing about this edge.

Upstream references