The end-to-end RAG pipeline¶
Two pipelines, ingest and query, each crossing the same small set of boundaries. Which boundaries are network hops depends on whether LocalRecall is a separate process or a linked library — and nothing else about the pipeline changes.
Ingest, standalone¶
Uploading guide.md to collection handbook on a chromem deployment.
POST /api/collections/handbook/upload,multipart/form-data, fieldfile. [network — client → localrecall :8080]- The handler copies the body to an
os.CreateTempfile, then renames it so the basename isguide.md, because the index key derives fromfilepath.Base. [in-process; fs write to /tmp] lookupCollection("handbook")resolves the*PersistentKB, constructing the engine now if a previous attempt left a nil placeholder. [in-process]PersistentKB.Storetakes the collection mutex, generates a UUID, and copies the file to<FILE_ASSETS>/handbook/<uuid>/guide.md. [fs write] The index key is<uuid>/guide.md.isChunkableFilechecks the lowercased extension..mdpasses. A file that fails here is stored and the request still returns200— see ingestion.fileToTextreads the file. Markdown is not parsed; the raw source is the text. [fs read] (An uppercase.MDfails here after passing step 5.)chunkFilesplits onstrings.Fieldsand accumulates greedily to 400 bytes with 0 overlap. [in-process]- Metadata is attached to every chunk:
type="file",source="<uuid>/guide.md",file_name="guide.md",created_at. Engine.StoreDocumentsembeds the chunks. On chromem this is one HTTP POST per chunk, fanned out acrossNumCPU()goroutines; on PostgreSQL it is one POST carrying every chunk. [network — localrecall → embeddings endpoint]- Vectors are persisted: one gzipped gob file per chunk under
<COLLECTION_DB_PATH>/<8-hex>/[fs write], or oneINSERTper chunk intodocuments_handbook[network — pgx → PostgreSQL]. 200 OKwith{filename, collection, key, created_at}. [network]
The mutex taken at step 4 is held across steps 5-10. Ingestion of one collection blocks searches of that same collection for its whole duration — including the embedding round trips.
Query, standalone¶
POST /api/collections/handbook/searchwith{"query": "...", "max_results": 8}. [network — client → localrecall :8080] Omittingmax_resultsgives you 5, or 1 if the collection holds fewer than 5 documents.PersistentKB.Searchtakes the collection mutex. [in-process]Engine.Searchembeds the query text. [network — localrecall → embeddings endpoint] Every search pays this; there is no query cache.- Nearest-neighbour lookup:
- chromem: brute-force scan of every chunk, dot product against the normalised query vector, bounded max-heap of size K. [in-process, no index]
- postgres: the RRF hybrid query, each arm pulling
max(K*10, 100)candidates from its own index, fused by rank. [network — pgx → PostgreSQL] - Results marshal as
types.Resultwith capitalised JSON keys.Similaritymeans different things on different engines — see retrieval. 200 OK. [network]
The caller now owns the rest of RAG: select which chunks to keep, assemble them into a prompt, and call the model. LocalRecall does none of that and has no knowledge of the model.
Embedded¶
The same eleven and six steps, with two edges deleted.
sequenceDiagram
participant U as user / agent
participant H as host process (local-ai)
participant E as embeddings endpoint
participant S as store (fs or PostgreSQL)
U->>H: HTTP POST (upload or agent turn)
Note over H: steps 2-8 run as Go calls inside this process
H->>H: in-process kb.Store(path, metadata)
H->>E: HTTP POST /embeddings
E-->>H: vectors
H->>S: fs write, or pgx INSERT
H-->>U: HTTP 200
U->>H: HTTP POST (query or agent turn)
H->>H: in-process kb.Search(query, k)
H->>E: HTTP POST /embeddings (query)
E-->>H: query vector
H->>S: scan (chromem, in-process) or hybrid SQL (pgx)
S-->>H: top-K chunks
H-->>U: HTTP 200 with results
Differences that matter:
| Standalone | Embedded | |
|---|---|---|
| Steps 1 and 11 of ingest | HTTP to :8080 |
Go method calls |
| Auth on collection operations | API_KEYS, all-or-nothing |
the host's own auth |
| The embeddings hop | present | still present |
| The PostgreSQL hop | present if selected | still present if selected |
| Raw-text ingest | impossible — no endpoint | trivial — write a temp file and call Store, which is exactly what LocalAGI does |
| Config source | 16 environment variables | constructor arguments supplied by the host |
The embeddings call never disappears. Embedding LocalRecall as a library removes the LocalRecall port, not the model dependency. In a LocalAI deployment that call is loopback HTTP from LocalAI to itself — a real HTTP request, subject to auth and visible in access logs.
Practical guidance¶
Chunk size¶
Decide before first ingest; changing it later requires a reset and re-upload, because the setting is process-global and existing chunks are never rewritten.
- The 400-byte default is roughly 60-100 English words. That is smaller than one paragraph, and with zero overlap a fact spanning a boundary survives in neither chunk.
- For English prose, a size in the low thousands of bytes with roughly 10-15% overlap is a defensible starting point.
- For non-ASCII corpora, scale up: the count is bytes, so CJK text gets about a third of the characters at the same setting.
- On chromem, larger chunks are also directly cheaper — fewer chunks means proportionally fewer HTTP requests during ingest.
Nothing here has been measured by us. Tune against your own queries.
Top-K¶
- Always send
max_results. The default of 5-or-1 counting documents is a trap on small collections. - With 400-byte chunks, each result is tiny; a K of 8-12 is closer to one paragraph of usable context than K of 3 would be with conventional chunk sizes.
- On PostgreSQL, K also drives candidate depth: each arm pulls
max(K*10, 100)rows before fusion, so cost grows faster than the result count. - Filter by rank position, never by an absolute similarity threshold. The score scale differs per engine and per index health.
When to move to PostgreSQL¶
Move when any of these is true:
| Signal | Why chromem cannot fix it |
|---|---|
| Query latency grows with corpus size | brute-force scan, no ANN index |
| Queries need exact identifiers, error codes, part numbers | chromem is dense-only; BM25 lives in PostgreSQL |
| Ingestion is dominated by embedding calls | chromem sends one HTTP request per chunk |
| More than one process must read the same collection | file-per-document store with process-local locking |
| You expect to change embedding models | chromem has no migration; PostgreSQL has a transactional one |
Stay on chromem when the corpus is a few thousand chunks, one process serves it, and you do not want a database to operate.
There is no in-place migration between engines. Moving means: create the
collection on the new engine, re-upload the originals from FILE_ASSETS (or
download them via .../raw first), and re-register external sources. Budget
embeddings capacity for a full re-embed of everything.
Before switching, confirm the target database has pg_textsearch,
shared_preload_libraries set, and a healthy idx_<table>_embedding — a
collection with no vector index works, silently, on sequential scans. See
storage.
Operating notes¶
- Persist
COLLECTION_DB_PATHandFILE_ASSETSunder every engine, database or not. - There is no health endpoint.
GET /api/collectionsproves only that the HTTP server and state directory are alive; it never touches the embeddings backend. - There is no graceful shutdown and no signal handling. Restarts during ingestion
can leave a file in the asset directory with no chunks in the store; the entry
will list and never match.
Repopulatefixes it, but only fires as a side effect of other operations. - Every restart re-fetches every external source immediately.
Upstream references¶
routes.go—uploadFileat 405-470,searchat 279-318,lookupCollectionat 137-156. Source-verified against v0.6.4, validated 2026-08-17.rag/persistency.go—Storeat 334-379 (mutex, UUID, copy, chunk, metadata),Searchat 185-190,chunkFileat 703-713. Validated 2026-08-17.rag/engine/chromem.go— per-chunk embedding at 82-89,NumCPUfan-out at 150. Validated 2026-08-17.rag/engine/postgres.go— whole-file batch embed at 681-686, hybrid query at 879-913, candidate constants at 851-859. Validated 2026-08-17.main.go— process-global chunk configuration at 72-88; no signal handling,e.Startat 92. Validated 2026-08-17.- LocalAGI
webui/collections/rag_provider.go— the embedded raw-text write path at 29-52. Validated against v2.9.0, 2026-08-17.