Repository navigation
fix: rune-safe byte slicing in Patch{Apply,SplitMax,AddContext} (fixes #132) - #155
Conversation
7407dfc to
fcc810a
Compare
|
Thanks for this, it does fix the #132 panic. Before merging I ran a round-trip test ( func TestPatchApplyMultibyteMonsterDelete(t *testing.T) {
a := "a日aaaaéaaaabaééaé日a日"
b := "a日aaabaaa日aaaaaaaaéaaaé"
dmp := New()
got, applied := dmp.PatchApply(dmp.PatchMake(a, b), a)
assert.Equal(t, []bool{true}, applied)
assert.Equal(t, b, got)
}With this PR, The cause is the This change fixed it for me: don't adjust if endLoc == -1 {
- text2End = alignRuneStart(text, int(math.Min(float64(startLoc+len(text1)), float64(len(text)))))
+ text2End = alignRuneEnd(text, int(math.Min(float64(startLoc+len(text1)), float64(len(text)))))
} else {
- endLoc = alignRuneStart(text, endLoc)
- text2End = alignRuneStart(text, int(math.Min(float64(endLoc+dmp.MatchMaxBits), float64(len(text)))))
+ text2End = alignRuneEnd(text, int(math.Min(float64(endLoc+dmp.MatchMaxBits), float64(len(text)))))
}// alignRuneEnd moves byte index idx forward to the next UTF-8 rune start.
func alignRuneEnd(s string, idx int) int {
for idx > 0 && idx < len(s) && !utf8.RuneStart(s[idx]) {
idx++
}
return idx
}With that change, the case above passes, the #132 repro still works, your tests pass, and 5,000 random round-trips match master. Could you add the test above along with the fix? Then I'm happy to merge this. |
Fixes sergi#132 (PatchApply slice bounds out of range on multi-byte text). PatchApply / PatchSplitMax / PatchAddContext slice text by UTF-8 byte index throughout. When MatchMain's Bitap returns a byte position that lands in the middle of a multi-byte rune (CJK 3 bytes, em dash 3 bytes, emoji 4 bytes), the resulting text[a:b] produces an invalid-UTF-8 string. Downstream calls (MatchMain on the slice, DiffMain, further text[c:d]) then either silently miss anchors or panic with "slice bounds out of range [:N] with length M". The reference JS diff-match-patch hits the same code paths but slices by UTF-16 code unit, where surrogate-pair-middle splits are well-defined — so the original algorithm tolerates mid-codepoint cuts in JS but not in Go. Fix: introduce a small `alignRuneStart(s, idx)` helper and call it before every byte-level slice that uses an index derived from MatchMain / DiffXIndex / arithmetic on multi-byte text. The alignment costs at most 1-3 bytes of shrinkage per cut point but guarantees the result is always valid UTF-8, so subsequent MatchMain / DiffMain operations behave predictably. Touched call sites: - PatchAddContext: prefix start, suffix end - PatchApply: startLoc after MatchMain, text2 end bound, replace end (perfect-match path), insert-at index, delete start/end (imperfect match path) - PatchSplitMax: diffText cut point (+ loop-progress guard so cutAt that aligns to 0 always advances by one full rune), remainder slice on bigpatch.Text, precontext start, postcontext end Tests added: - TestAlignRuneStart: byte-index → rune-start mapping (ASCII + multi-byte cases). - TestPatchApply_Issue132_NoPanic: the minimal repro from the upstream issue no longer panics. - TestPatchSplitMax_NoInvalidUTF8: pure-CJK PatchMake + PatchSplitMax produces only valid-UTF-8 diff text. Full existing test suite still passes; no behavior change on ASCII-only input (all alignment calls are no-ops when the index already sits at a rune boundary or at len(s)).
Review follow-up. - PatchApply: round the end of text2 forward instead of moving endLoc back. Rounding back dropped the last rune from text2, so on the long-delete path a deletion of that rune silently did nothing while the patch still reported success (TestPatchApplyMultibyteMonsterDelete, from review). - PatchAddContext: round the suffix end forward for the same reason. A shortened suffix made PatchAddPadding treat the patch as touching the end of the text, so PatchApply fell back to a fuzzy match and could place an insertion one rune early. - PatchApply: clamp the insert/delete indices and the padding cut points to the current text length, as substring does in the reference implementation. Earlier edits in a patch can leave the text shorter than indices computed from text2. This panicked with "slice bounds out of range" on both ASCII and multi-byte text when a patch was applied to a changed text. - PatchApply: when stripping the padding, drop a rune that straddles a cut point so the result stays valid UTF-8. - Add a test for the rune-aligned cut in diffText1Prefix. Checked with randomized tests against master, 100k cases each: - PatchMake/PatchApply round-trips on multi-byte text: all correct, as on master. - Patches applied to a modified multi-byte text: master panics 623 times and returns invalid UTF-8 22485 times; this branch does neither. - ASCII: identical output to master wherever master does not panic (master panics 81 times when applying to a modified text).
fcc810a to
ce0771e
Compare
|
Thanks for the careful review and the repro, that was a real bug. I rebased onto master (the only conflict was the new
Randomized checks against master, 100k cases each (alphabet includes
Each new test fails without its fix. Happy to split the clamping into a separate PR if you'd rather keep this one narrower. |
There was a problem hiding this comment.
Ok this addresses everything from my review, and the extra fixes turned out to be worth it. Verified and approving.
What I checked
- Full suite passes (also with
-race),go vetandgofmtclean; the branch is on top of current master. - Checked out
b2f00cc(the pre-review state):TestPatchApplyMultibyteMonsterDeletefails there and passes here, so the reported bug and its fix are both confirmed. All four tests added in the follow-up fail without their fixes. - The
text2/PatchAddContextend handling now rounds outward exactly as discussed, andendLocis left alone. - The clamping matches the JS reference's
substringsemantics on every path I traced, including theendIndex < startIndexguard and the padding strip when deletions eat into the padding (thelen(text) < 2*len(nullPadding)case panics on master). - The
PatchSplitMaxprogress guard is correct and necessary: with a cut collapsed to 0 the remainder slice would be a no-op and the loop would spin forever.
Independent differential fuzz vs master (same deterministic seed on both branches, 250k (a,b) pairs → 500k applications, three alphabets: ASCII, your é日—🚀 set, and a nastier one with combining marks, astral chars, and shared-byte-prefix runes):
| master | this branch | |
|---|---|---|
round-trip PatchMake → PatchApply |
all correct | all correct; output byte-identical to master in 100% of cases |
| apply to modified text, ASCII (shrink-biased mutator) | 51 panics | 0 panics; output identical to master everywhere else |
| apply to modified text, multi-byte | 510 + 149 panics, ~12k invalid UTF-8 | 0 panics, 0 invalid UTF-8 |
Two minor nits, no need to hold the merge for them — happy to take them as a follow-up:
isValidUTF8/utf8DecodeFirstin the test file hand-rolls UTF-8 validation and is weaker thanutf8.Valid(continuation bytes aren't checked, so e.g."\xc0\x80"or"\xe4\xb8A"pass). It's a_test.gofile, so importingunicode/utf8is free —utf8.ValidStringwould be both shorter and stricter.TestPatchSplitMax_NoInvalidUTF8skips when no patches are generated; a PatchMake regression could silently neuter it. Asserting non-empty would be safer.
Thanks again for the thorough work!
Fixes #132 (
PatchApplyslice bounds out of range).Root cause
PatchApply/PatchSplitMax/PatchAddContextslice text by UTF-8 byte index. WhenMatchMainreturns a byte position inside a multi-byte rune,text[a:b]produces invalid UTF-8, and later slices can run past the end of the text and panic. The JS reference slices by UTF-16 code unit and itssubstringclamps out-of-range indices, so the same algorithm does not fail there.Fix
alignRuneStart/alignRuneEndmove a byte index to a rune boundary. The start of a range is rounded back and the end is rounded forward, so a range never loses a rune it should contain.PatchAddContext: prefix start, suffix end.PatchApply:startLoc, the end oftext2, insert and delete positions.PatchSplitMax: thediffTextcut point (with a progress guard), the remainder slice, precontext start, postcontext end (diffText1Prefix).PatchApplyclamps insert/delete indices and the padding cut points to the current text length, matchingsubstringin the reference. Earlier edits in a patch can leave the text shorter than indices computed fromtext2; this used to panic on ASCII input too.Tests
TestAlignRuneStart,TestAlignRuneEnd,TestDiffText1PrefixRuneAlignedTestPatchApply_Issue132_NoPanic: the repro from PatchApply panics with slice bounds out of range #132.TestPatchApplyMultibyteMonsterDelete: from review, long-delete path.TestPatchMakeApplyContextEndsInsideRune: suffix context ending inside a rune.TestPatchApplyStripPaddingKeepsValidUTF8,TestPatchApplyChangedTextNoPanicTestPatchSplitMax_NoInvalidUTF8Randomized comparison with master, 100k cases each: multi-byte round-trips are all correct on both; applying patches to a modified multi-byte text goes from 623 panics and 22485 invalid-UTF-8 results to none; ASCII output is identical to master wherever master does not panic (master panics 81 times when applying to a modified text, this branch never does).