Skip to content

fix: rune-safe byte slicing in Patch{Apply,SplitMax,AddContext} (fixes #132) - #155

Merged
sergi merged 2 commits into
sergi:masterfrom
SnowWarri0r:pr1/fix-issue-132-rune-safe-slicing
Nov 17, 2026
Merged

sergi merged 2 commits into
sergi:masterfrom
SnowWarri0r:pr1/fix-issue-132-rune-safe-slicing

Conversation

@SnowWarri0r

@SnowWarri0r SnowWarri0r commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #132 (PatchApply slice bounds out of range).

Root cause

PatchApply / PatchSplitMax / PatchAddContext slice text by UTF-8 byte index. When MatchMain returns a byte position inside a multi-byte rune, text[a:b] produces invalid UTF-8, and later slices can run past the end of the text and panic. The JS reference slices by UTF-16 code unit and its substring clamps out-of-range indices, so the same algorithm does not fail there.

Fix

  • alignRuneStart / alignRuneEnd move a byte index to a rune boundary. The start of a range is rounded back and the end is rounded forward, so a range never loses a rune it should contain.
    • PatchAddContext: prefix start, suffix end.
    • PatchApply: startLoc, the end of text2, insert and delete positions.
    • PatchSplitMax: the diffText cut point (with a progress guard), the remainder slice, precontext start, postcontext end (diffText1Prefix).
  • PatchApply clamps insert/delete indices and the padding cut points to the current text length, matching substring in the reference. Earlier edits in a patch can leave the text shorter than indices computed from text2; this used to panic on ASCII input too.
  • When the padding is stripped, a rune straddling a cut point is dropped so the result is valid UTF-8.

Tests

  • TestAlignRuneStart, TestAlignRuneEnd, TestDiffText1PrefixRuneAligned
  • TestPatchApply_Issue132_NoPanic: the repro from PatchApply panics with slice bounds out of range #132.
  • TestPatchApplyMultibyteMonsterDelete: from review, long-delete path.
  • TestPatchMakeApplyContextEndsInsideRune: suffix context ending inside a rune.
  • TestPatchApplyStripPaddingKeepsValidUTF8, TestPatchApplyChangedTextNoPanic
  • TestPatchSplitMax_NoInvalidUTF8

Randomized comparison with master, 100k cases each: multi-byte round-trips are all correct on both; applying patches to a modified multi-byte text goes from 623 panics and 22485 invalid-UTF-8 results to none; ASCII output is identical to master wherever master does not panic (master panics 81 times when applying to a modified text, this branch never does).

@sergi

sergi commented Nov 16, 2026

Copy link
Copy Markdown
Owner

Thanks for this, it does fix the #132 panic. Before merging I ran a round-trip test (PatchMake then PatchApply on random multibyte text), and it found a case that master handles correctly but this PR doesn't:

func TestPatchApplyMultibyteMonsterDelete(t *testing.T) {
	a := "a日aaaaéaaaabaééaé日a日"
	b := "a日aaabaaa日aaaaaaaaéaaaé"
	dmp := New()
	got, applied := dmp.PatchApply(dmp.PatchMake(a, b), a)
	assert.Equal(t, []bool{true}, applied)
	assert.Equal(t, b, got)
}

With this PR, got keeps a trailing 日 that should have been deleted, and applied still reports [true], so the failure is silent.

The cause is the endLoc = alignRuneStart(text, endLoc) line in PatchApply. The text being replaced is over 32 bytes here, so the long-delete path runs even though it's only 20 characters. Moving endLoc back to a rune start also moves the end of text2 (endLoc + MatchMaxBits) back, so text2 loses its last rune and the deletion has nothing to remove.

This change fixed it for me: don't adjust endLoc, and round the end of text2 forward to the next rune start, so the slice includes the whole last rune instead of dropping it:

 			if endLoc == -1 {
-				text2End = alignRuneStart(text, int(math.Min(float64(startLoc+len(text1)), float64(len(text)))))
+				text2End = alignRuneEnd(text, int(math.Min(float64(startLoc+len(text1)), float64(len(text)))))
 			} else {
-				endLoc = alignRuneStart(text, endLoc)
-				text2End = alignRuneStart(text, int(math.Min(float64(endLoc+dmp.MatchMaxBits), float64(len(text)))))
+				text2End = alignRuneEnd(text, int(math.Min(float64(endLoc+dmp.MatchMaxBits), float64(len(text)))))
 			}
// alignRuneEnd moves byte index idx forward to the next UTF-8 rune start.
func alignRuneEnd(s string, idx int) int {
	for idx > 0 && idx < len(s) && !utf8.RuneStart(s[idx]) {
		idx++
	}
	return idx
}

With that change, the case above passes, the #132 repro still works, your tests pass, and 5,000 random round-trips match master. Could you add the test above along with the fix? Then I'm happy to merge this.

Fixes sergi#132 (PatchApply slice bounds out of range on multi-byte text).

PatchApply / PatchSplitMax / PatchAddContext slice text by UTF-8 byte
index throughout. When MatchMain's Bitap returns a byte position that
lands in the middle of a multi-byte rune (CJK 3 bytes, em dash 3 bytes,
emoji 4 bytes), the resulting text[a:b] produces an invalid-UTF-8
string. Downstream calls (MatchMain on the slice, DiffMain, further
text[c:d]) then either silently miss anchors or panic with
"slice bounds out of range [:N] with length M".

The reference JS diff-match-patch hits the same code paths but slices
by UTF-16 code unit, where surrogate-pair-middle splits are
well-defined — so the original algorithm tolerates mid-codepoint cuts
in JS but not in Go.

Fix: introduce a small `alignRuneStart(s, idx)` helper and call it
before every byte-level slice that uses an index derived from
MatchMain / DiffXIndex / arithmetic on multi-byte text. The alignment
costs at most 1-3 bytes of shrinkage per cut point but guarantees the
result is always valid UTF-8, so subsequent MatchMain / DiffMain
operations behave predictably.

Touched call sites:
  - PatchAddContext: prefix start, suffix end
  - PatchApply: startLoc after MatchMain, text2 end bound, replace end
    (perfect-match path), insert-at index, delete start/end (imperfect
    match path)
  - PatchSplitMax: diffText cut point (+ loop-progress guard so cutAt
    that aligns to 0 always advances by one full rune), remainder
    slice on bigpatch.Text, precontext start, postcontext end

Tests added:
  - TestAlignRuneStart: byte-index → rune-start mapping (ASCII +
    multi-byte cases).
  - TestPatchApply_Issue132_NoPanic: the minimal repro from the
    upstream issue no longer panics.
  - TestPatchSplitMax_NoInvalidUTF8: pure-CJK PatchMake + PatchSplitMax
    produces only valid-UTF-8 diff text.

Full existing test suite still passes; no behavior change on
ASCII-only input (all alignment calls are no-ops when the index already
sits at a rune boundary or at len(s)).
Review follow-up.

- PatchApply: round the end of text2 forward instead of moving endLoc
  back. Rounding back dropped the last rune from text2, so on the
  long-delete path a deletion of that rune silently did nothing while the
  patch still reported success (TestPatchApplyMultibyteMonsterDelete,
  from review).
- PatchAddContext: round the suffix end forward for the same reason. A
  shortened suffix made PatchAddPadding treat the patch as touching the
  end of the text, so PatchApply fell back to a fuzzy match and could
  place an insertion one rune early.
- PatchApply: clamp the insert/delete indices and the padding cut points
  to the current text length, as substring does in the reference
  implementation. Earlier edits in a patch can leave the text shorter
  than indices computed from text2. This panicked with "slice bounds out
  of range" on both ASCII and multi-byte text when a patch was applied to
  a changed text.
- PatchApply: when stripping the padding, drop a rune that straddles a
  cut point so the result stays valid UTF-8.
- Add a test for the rune-aligned cut in diffText1Prefix.

Checked with randomized tests against master, 100k cases each:
- PatchMake/PatchApply round-trips on multi-byte text: all correct, as on
  master.
- Patches applied to a modified multi-byte text: master panics 623 times
  and returns invalid UTF-8 22485 times; this branch does neither.
- ASCII: identical output to master wherever master does not panic
  (master panics 81 times when applying to a modified text).
@SnowWarri0r
SnowWarri0r force-pushed the pr1/fix-issue-132-rune-safe-slicing branch from fcc810a to ce0771e Compare November 17, 2026 06:13
@SnowWarri0r

Copy link
Copy Markdown
Contributor Author

Thanks for the careful review and the repro, that was a real bug.

I rebased onto master (the only conflict was the new diffText1Prefix in PatchSplitMax; it now cuts at a rune start) and added a follow-up commit:

  • Your fix: text2 now ends at alignRuneEnd(...) and endLoc is left alone. TestPatchApplyMultibyteMonsterDelete is in, as written.
  • Same pattern in PatchAddContext: the suffix end was also rounded back, which could shorten the context below the padding length. PatchApply then padded the patch as if it touched the end of the text, fell back to a fuzzy match, and put the insertion one rune early while still reporting success, e.g. " aba— 🚀" → " a aba—a 🚀". It now rounds forward too.
  • Out-of-range indices in PatchApply: when a patch is applied to a text that has changed, earlier edits can leave the text shorter than the insert/delete indices computed from text2, and the padding strip can run past the end. Master panics here on ASCII as well. These are now clamped the way substring clamps in the JS reference.
  • Padding strip: an insertion that lands in the padding could leave half a rune at the cut. It now drops the straddling rune.

Randomized checks against master, 100k cases each (alphabet includes é, 日, —, 🚀):

master this branch
multi-byte PatchMake → PatchApply round-trip all correct all correct
multi-byte patch applied to a modified text 623 panics, 22485 invalid UTF-8 0 / 0
ASCII patch applied to a modified text 81 panics 0
ASCII output where master doesn't panic identical

Each new test fails without its fix. Happy to split the clamping into a separate PR if you'd rather keep this one narrower.

@sergi sergi left a comment •

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok this addresses everything from my review, and the extra fixes turned out to be worth it. Verified and approving.

What I checked

  • Full suite passes (also with -race), go vet and gofmt clean; the branch is on top of current master.
  • Checked out b2f00cc (the pre-review state): TestPatchApplyMultibyteMonsterDelete fails there and passes here, so the reported bug and its fix are both confirmed. All four tests added in the follow-up fail without their fixes.
  • The text2 / PatchAddContext end handling now rounds outward exactly as discussed, and endLoc is left alone.
  • The clamping matches the JS reference's substring semantics on every path I traced, including the endIndex < startIndex guard and the padding strip when deletions eat into the padding (the len(text) < 2*len(nullPadding) case panics on master).
  • The PatchSplitMax progress guard is correct and necessary: with a cut collapsed to 0 the remainder slice would be a no-op and the loop would spin forever.

Independent differential fuzz vs master (same deterministic seed on both branches, 250k (a,b) pairs → 500k applications, three alphabets: ASCII, your é日—🚀 set, and a nastier one with combining marks, astral chars, and shared-byte-prefix runes):

master this branch
round-trip PatchMake → PatchApply all correct all correct; output byte-identical to master in 100% of cases
apply to modified text, ASCII (shrink-biased mutator) 51 panics 0 panics; output identical to master everywhere else
apply to modified text, multi-byte 510 + 149 panics, ~12k invalid UTF-8 0 panics, 0 invalid UTF-8

Two minor nits, no need to hold the merge for them — happy to take them as a follow-up:

  • isValidUTF8/utf8DecodeFirst in the test file hand-rolls UTF-8 validation and is weaker than utf8.Valid (continuation bytes aren't checked, so e.g. "\xc0\x80" or "\xe4\xb8A" pass). It's a _test.go file, so importing unicode/utf8 is free — utf8.ValidString would be both shorter and stricter.
  • TestPatchSplitMax_NoInvalidUTF8 skips when no patches are generated; a PatchMake regression could silently neuter it. Asserting non-empty would be safer.

Thanks again for the thorough work!

@sergi
sergi merged commit 7f6d802 into sergi:master Nov 17, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PatchApply panics with slice bounds out of range

2 participants