chunk_document panics on multibyte text (byte-slice cuts a char) #8

Summary: tail_overlap in crates/embeddings/src/chunker.rs slices text[text.len() - n..] using byte offsets. On multibyte text (CJK, Greek, emoji), that offset can land inside a multibyte character, which panics with "byte index N is not a char boundary." This is reachable from the public chunk_document API whenever a chunk flush occurs on multibyte text — i.e. any user document in a non-Latin script (or with emoji) long enough to need more than one chunk.

Severity: S2 — user-facing broken flow. A user uploading/editing a CJK, Greek, Cyrillic, Arabic, or emoji-containing document long enough to span multiple chunks (default chunk_size = 2048) will crash the indexing/embedding pipeline for that document; the document itself likely persists, but it will not be searchable via semantic/hybrid search and the import/index job will fail.

Environment:

• Deploy: local / any (code-level bug)
• Commit: 6411ddb
• User / role: any user editing or importing a document in a non-Latin script

Repro steps:

1. Direct function-level repro (guaranteed panic): call tail_overlap(&"α".repeat(10), 5).
   • "α".repeat(10) is 20 bytes / 10 chars. start = 20 - 5 = 15, which falls inside the 8th α (occupying bytes 14..16) — not a char boundary.
2. End-to-end repro path: call chunk_document(title, body, config) with a body containing multiple paragraphs of multibyte text whose cumulative length exceeds config.chunk_size, so a mid-document flush triggers tail_overlap on accumulated multibyte content.

Expected: Chunking multibyte text produces valid UTF-8 chunks with no panic, regardless of where the byte-length budget falls relative to character boundaries.

Observed: Panic: byte index 15 is not a char boundary; it is inside 'α' (bytes 14..16) of the string.

Evidence:

• Code: crates/embeddings/src/chunker.rs:149-158 (tail_overlap), specifically text[start..] / text[text.len() - n..] on raw byte indices with no char_indices/is_char_boundary guard.
• Existing test unicode_body_within_single_chunk_is_preserved (crates/embeddings/src/chunker.rs:385-386) only covers the single-chunk (no-flush) path and does not exercise tail_overlap, so it does not catch this.

Suspected area: crates/embeddings/src/chunker.rs::tail_overlap — needs a char-boundary-safe tail slice (e.g. iterate char_indices() from the end, or use floor_char_boundary once stable / a manual boundary search).

Findings consolidated: This ticket bundles the primary panic with three secondary quality issues in the same function/module, all surfaced by the same review pass. They do not share a line-level root cause with the panic — bundling them here is for filing efficiency, not because they're the same bug. If any is worth independent tracking, split it out:

1. No separator at the overlap seam. current = format!("{overlap_text}{para}") (line ~95) and text.push_str(&word_chunk) (line ~73) concatenate directly, merging the last word of the overlap with the first word of the next paragraph/word-chunk with no space or newline.
2. Computed overlap silently disappears after an oversized-paragraph flush. The else branch (lines ~96-101) that handles the non-oversized case doesn't carry overlap_text forward the way the oversized-paragraph branch does — chunks immediately following an oversized-paragraph split lose their intended overlap.
3. Doc/code drift. ChunkerConfig.chunk_size doc comment (line 14) says "characters," but every length check in the function (para.len(), current.len(), text[start..]) is a byte count via str::len().

Scope expansion suggested: A dedicated boundary-exploration pass on the chunker with a matrix of scripts (CJK, RTL/Arabic, emoji with combining/ZWJ sequences) would likely surface more of the same class of bug.