Skip to content

Chunker overlap seams: missing separator and dropped overlap after oversized-paragraph flushes #19

Closed

Description

@jjbaumert

Summary: Split from #8 (bundled items 1–2, deferred there because they change chunk output that existing tests pin — a deliberate behavior change, not part of the panic fix).

  1. No separator at the overlap seam. chunk_document joins overlap text and the following content directly — format!("{overlap_text}{para}") (crates/embeddings/src/chunker.rs, flush branch) and text.push_str(&word_chunk) (word-split loop) — merging the overlap's last word with the next content's first word at every chunk seam. Confirmed output shapes like "delta echofoxtrot golf". Degrades embedding quality at every seam.
  2. Computed overlap silently dropped after an oversized-paragraph flush. The non-oversized else branch never carries overlap_text forward when starting an empty accumulator, so chunks immediately following an oversized-paragraph split lose their intended overlap entirely.

Severity: S4 — no crash, no data loss; systematic quality degradation of embeddings at chunk boundaries.

Environment: any; found at commit 6411ddb, still present at da44362 (the #8 panic fix deliberately did not alter output).

Expected: Overlap text joins the next content at a word boundary (space/newline), and overlap survives across oversized-paragraph flushes.

Observed: Words merge at seams; overlap disappears after word-split paragraphs.

Suspected area: crates/embeddings/src/chunker.rs::chunk_document. Note for the fix: chunk output changes, so the pinned expectations (notably the chunk_size + overlap ceiling proptest and seam tests) must be updated with the change as a deliberate behavior change, and re-indexing existing documents may be warranted for consistency.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingverification-findingFinding from a verification run

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions