Repository navigation
perf(csv-parse): append ordinary bytes without the full token checks - #561
Open
deadstrobe5 wants to merge 1 commit into
Open
deadstrobe5 wants to merge 1 commit into
deadstrobe5 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every input byte goes through the escape, quote, record delimiter, comment, delimiter, max record size and trim checks before it is appended to the current field. Most bytes of a real file are ordinary field content that can't start any of those tokens. This builds a 256-entry table once per
parsecall that marks the first byte of every delimiter, record delimiter, quote, escape and comment, plus CR, LF and the trim characters whenltrimorrtrimis set. A byte that isn't marked is appended directly, as long as the parser isn't escaping or inside a comment and, withrtrim, isn't right after a closing quote. Every marked byte still goes through the existing code. Withmax_record_sizethe old path is kept unchanged.columns: true, trim: truecolumns: truecolumns: true, cast: trueCPU time, median of 5 to 7 runs alternating old and new code in the same process, Node 24.22.2. The machine is shared, so an A/A run of the same code varied by up to 7%. To compare outputs I ran 40k random inputs with random options (multi-byte delimiters, quotes, escapes, comments, record delimiters, trim, raw, info, BOM, utf16le and latin1, from/to, relax options, columns) through the transform API, split into random chunks down to one byte. Records, info and errors were identical. Existing tests pass.