You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Chunker overlap seams: missing separator and dropped overlap after oversized-paragraph flushes #19
Summary: Split from #8 (bundled items 1–2, deferred there because they change chunk output that existing tests pin — a deliberate behavior change, not part of the panic fix).
No separator at the overlap seam.chunk_document joins overlap text and the following content directly — format!("{overlap_text}{para}") (crates/embeddings/src/chunker.rs, flush branch) and text.push_str(&word_chunk) (word-split loop) — merging the overlap's last word with the next content's first word at every chunk seam. Confirmed output shapes like "delta echofoxtrot golf". Degrades embedding quality at every seam.
Computed overlap silently dropped after an oversized-paragraph flush. The non-oversized else branch never carries overlap_text forward when starting an empty accumulator, so chunks immediately following an oversized-paragraph split lose their intended overlap entirely.
Severity: S4 — no crash, no data loss; systematic quality degradation of embeddings at chunk boundaries.
Environment: any; found at commit 6411ddb, still present at da44362 (the #8 panic fix deliberately did not alter output).
Expected: Overlap text joins the next content at a word boundary (space/newline), and overlap survives across oversized-paragraph flushes.
Observed: Words merge at seams; overlap disappears after word-split paragraphs.
Suspected area:crates/embeddings/src/chunker.rs::chunk_document. Note for the fix: chunk output changes, so the pinned expectations (notably the chunk_size + overlap ceiling proptest and seam tests) must be updated with the change as a deliberate behavior change, and re-indexing existing documents may be warranted for consistency.
Activity