Skip to content

GH-1386: Zero-pad inline views during split and transfer - #1387

Open
hutiefang76 wants to merge 1 commit into
apache:mainfrom
hutiefang76:frank/arrow1386-view-padding-20261024
Open

hutiefang76 wants to merge 1 commit into
apache:mainfrom
hutiefang76:frank/arrow1386-view-padding-20261024

Conversation

@hutiefang76

Copy link
Copy Markdown
Contributor

What's Changed

splitAndTransfer allocates a new view buffer but only writes the length and payload of inline values. When the allocator returns non-zero memory, the remaining bytes violate the view layout's zero-padding requirement. Clear just the unused inline bytes after copying the payload; the long-value path is unchanged.

Add deterministic regressions for UTF-8 and binary views using buffers prefilled with 0xff. They cover lengths 0, 1, 11, 12 and 13, nulls, a non-zero split offset, source preservation and allocator cleanup. Both new tests fail against the unmodified implementation with an unexpected 0xff padding byte.

Validation on JDK 21:

  • mvn -B -ntp -Dmaven.gitcommitid.nativegit=true -pl vector -am verify passed. The vector module ran 1193 cases in each of its two configured allocator executions, with 4 skipped per execution and no failures or errors.
  • The full TestVariableWidthViewVector class passed all 59 cases in each execution, with no skips.
  • Spotless and Checkstyle passed. The native Git option is needed locally for the existing Git plugin's worktree support; no build configuration was changed.

Closes #1386.

AI-generated PR description. Source investigation, implementation and automated review assisted by Codex and DeepSeek.

Clear the unused inline bytes in each transferred view rather than relying on
newly allocated memory being zeroed. Exercise text and binary views with
deterministically non-zero allocations and boundary-length values.

Generated-by: Codex and DeepSeek
@apache apache deleted a comment from hutiefang76 Nov 6, 2026
@jbonofre jbonofre added the bug-fix PRs that fix a big. label Nov 6, 2026
@apache apache deleted a comment from github-actions Bot Nov 6, 2026
@github-actions github-actions Bot added this to the 20.0.0 milestone Nov 6, 2026
@hutiefang76

Copy link
Copy Markdown
Contributor Author

I checked the logs from the November 6 rerun at bde43f264218298ddfd6da470aa8cb460180778d:

  • pre-commit: cmake-format raises ValueError: Cannot use capturing groups in re.Scanner under Python 3.15.2 while processing adapter/orc/CMakeLists.txt and gandiva/CMakeLists.txt. The other hooks passed.
  • macOS JNI: compilation stops in _deps/protobuf-src/src/google/protobuf/parse_context.h with fatal error: 'utf8_validity.h' file not found while building Gandiva.

These errors occur in tooling/native build paths outside this PR's two Java vector files. I have not reproduced them locally or established a dependency fix; the checks remain failed.

AI-generated CI log summary, checked against the linked job logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug-fix PRs that fix a big.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Java] splitAndTransfer of a view vector leaves non-zero padding in inlined views

2 participants