Chunking¶
One strategy, 191 lines, no configuration beyond two numbers. Extracted text is split into fixed-size pieces measured in bytes, aligned to word boundaries, with optional overlap.
Defaults: 400 bytes, 0 overlap. Both are unusually small compared with mainstream RAG defaults, and both are process-global.
The algorithm¶
pkg/chunk/chunking.go, entered from chunkFile (rag/persistency.go:703-713):
opts := chunk.Options{MaxSize: maxchunksize, Overlap: chunkOverlap, SplitLongWords: true}
chunks := chunk.SplitParagraphIntoChunksWithOptions(content, opts)
Step by step:
strings.Fields(paragraph)splits the whole document on any whitespace.- Words are appended greedily while
currentChunk.Len() + 1 + len(word) <= MaxSize. - On overflow the chunk is flushed and a new one starts, optionally prefixed with an overlap tail.
- A word longer than
MaxSizeis hard-split intoMaxSize-byte pieces whenSplitLongWordsis true — it is hardcoded true.
It is not recursive, not separator-aware, and not token-based. There is no sentence detection, no heading awareness, no markdown parsing, no code-block protection.
The function name is wrong¶
SplitParagraphIntoChunks does no paragraph handling whatsoever.
strings.Fields is the reason. It treats every run of whitespace — spaces,
single newlines, blank lines, tabs, indentation — identically, and the words are
re-joined with a single space. Everything structural is gone before the first
chunk boundary is chosen:
| Input structure | After strings.Fields |
|---|---|
Paragraph breaks (\n\n) |
one space |
| Markdown heading on its own line | inline text, # retained as a token |
| Markdown table rows | one run-on line of |-separated cells |
| Code indentation | collapsed |
PDF page separators (\n\n, inserted by the extractor) |
one space |
| List items | run together |
The project's own test acknowledges the gap: chunk_test.go:45-49 is named
"should split on paragraph boundaries when possible" and asserts only
ToNot(BeEmpty()). It does not test paragraph behaviour, because there is none.
Two practical consequences:
- A chunk boundary can land anywhere, including mid-sentence and between a heading and the text it introduces. At 400 bytes with zero overlap, a heading frequently ends up in a different chunk from its own content — so a query matching the heading retrieves a fragment that does not contain the answer, and a query matching the answer retrieves a fragment with no indication of what it is about.
- Markdown syntax is embedded as content.
#,|,```and link brackets all consume the byte budget and all get embedded. For heavily formatted documents a meaningful fraction of every chunk is punctuation.
Bytes, not characters¶
MaxSize is compared against len() on a Go string, which is bytes.
The effect on a multilingual corpus is direct, and this matters because the
ecosystem's default embedding model is multilingual
(granite-embedding-107m-multilingual):
| Script | Bytes per character (UTF-8) | Effective characters in a 400-byte chunk |
|---|---|---|
| ASCII English | 1 | ~400 |
| Latin with accents, Greek, Cyrillic | 2 | ~200 |
| CJK, Devanagari | 3 | ~133 |
| Emoji, some symbols | 4 | ~100 |
A Japanese or Chinese corpus therefore gets chunks roughly one third the
semantic size of an English one at the same setting. If your corpus is not
predominantly ASCII, raising MAX_CHUNKING_SIZE proportionally is the first
tuning move to make.
There is a related sharp edge: splitLongString slices bytes, so hard-splitting
a long non-ASCII word can split a multi-byte rune and produce invalid UTF-8
inside a chunk. The PDF path and the PostgreSQL insert path both sanitise invalid
UTF-8 afterwards, so this is mitigated for PDFs and for PostgreSQL — but not
for a .txt or .md file going into chromem.
Overlap¶
Zero by default. When set, overlapTail walks words backwards from the
just-flushed chunk, accumulating whole words plus their spaces while the running
length stays within the overlap budget, and prepends the result to the next
chunk.
Two properties worth knowing:
- Overlap is word-aligned and never exceeds the configured bytes.
- It is best-effort: the tail is prepended only if it plus the next word still
fits within
MaxSize. Otherwise it is dropped silently. You cannot rely on every boundary having overlap.
Clamping: Overlap >= MaxSize becomes MaxSize - 1; a negative value becomes 0.
Overlap is a recent feature (PR #41, shipped around v0.6.0). Its default of zero is the single most consequential retrieval-quality setting in LocalRecall, because with 400-byte chunks and no overlap, any fact spanning a boundary exists in neither chunk in complete form.
Configuration¶
| Knob | Default | Set by | Notes |
|---|---|---|---|
| chunk size | 400 bytes | MAX_CHUNKING_SIZE |
parse error → process exits at startup |
| overlap | 0 bytes | CHUNK_OVERLAP |
parse error → process exits at startup |
SplitLongWords |
true |
— | hardcoded, not configurable |
Both are parsed with strconv.Atoi and call e.Logger.Fatal on failure, so
MAX_CHUNKING_SIZE=400k kills the process rather than falling back. There is no
range validation: 0 is accepted, then clamped to 1 inside the chunker,
producing one chunk per byte.
Changing the configuration¶
The values are process-global, captured once at startup and baked into every
PersistentKB. There is no per-collection chunk configuration anywhere in the
API or the library constructors.
Changing them does not re-chunk anything that already exists. A collection ingested at 400 bytes keeps 400-byte chunks after you set 1200. Two ways to apply new settings:
- Reset and re-upload. Note that reset is really delete — see
collections, route 8 — so you need
your originals elsewhere, or you must download them via
.../rawfirst. - Trigger a
Repopulate. It re-chunks and re-embeds every chunkable file on disk using the current settings. It is not exposed as an endpoint; it fires on an embedding-dimension mismatch at collection open, and on entry removal whenLOCALRECALL_REPOPULATE_DELETE=true. Both are side effects of other operations, so treat this as a mechanism to be aware of rather than a tool to reach for.
What size should you use?¶
There is no measured recommendation here — nothing in this chapter was executed. What the code establishes is the shape of the trade-off:
| Setting | Effect | Cost |
|---|---|---|
Larger MAX_CHUNKING_SIZE |
more context per hit; fewer boundary-straddling facts; fewer embedding calls (materially cheaper on chromem, which sends one HTTP request per chunk) | each vector represents more, diluting topical specificity; more of the context window consumed per retrieved chunk |
Non-zero CHUNK_OVERLAP |
facts spanning a boundary survive in at least one chunk | duplicate text inflates the store and can return near-identical hits, wasting top-K slots |
| Default 400 / 0 | maximal precision per vector; smallest possible retrieval unit | ~60-100 English words per chunk, ~100 tokens — often less than one paragraph |
A defensible starting point for prose in English is a chunk size in the low thousands of bytes with an overlap of roughly 10-15% of it, then measuring. For non-ASCII corpora, scale the byte figure by the bytes-per-character table above. Whatever you pick, decide it before first ingest — retrofitting means a reset.
One detail to keep in mind while tuning top-K: GET .../entries/:entry
deliberately re-extracts text from the file on disk rather than concatenating
chunks, precisely so overlap does not appear duplicated. So a healthy-looking
entry-content response tells you nothing about your chunk settings. Use
chunk_count from that same response for that.
Edge case: the empty file¶
SplitParagraphIntoChunks("") returns []string{""} — a one-element slice
holding an empty string, not an empty slice.
PersistentKB.store guards with len(pieces) == 0 → "no chunks generated for
file". An empty file produces one chunk, so it passes the guard, and an empty
string is sent to the embedder and stored as a vector. The same applies to a
scanned PDF with no text layer. Neither errors; both create an entry that
occupies a top-K slot and means nothing.
Upstream references¶
pkg/chunk/chunking.go—Optionsat 8-16, clamping at 69-78,strings.Fieldsat 99, greedy accumulation at 129-147, overlap prepend at 149-157,splitLongStringat 20-34, empty-input return at 82-84. Source-verified against v0.6.4, validated 2026-08-17.pkg/chunk/chunk_test.go— the empty-input expectation at 20-25 and the non-assertion at 45-49. Validated 2026-08-17.rag/persistency.go—chunkFileat 703-713 withSplitLongWords: true;Repopulateat 249; theno chunks generatedguard at 450-452. Validated 2026-08-17.main.go— defaultschunkingSize := 400at 72 andoverlap := 0at 81; fatal parse handling at 73-88. Validated 2026-08-17.- LocalRecall PR #41 — chunk overlap and entry fetch. Validated 2026-08-17.