Skip to content

perf(csv-parse): append ordinary bytes without the full token checks - #561

Open
deadstrobe5 wants to merge 1 commit into
adaltas:masterfrom
deadstrobe5:perf/parse-plain-bytes
Open

deadstrobe5 wants to merge 1 commit into
adaltas:masterfrom
deadstrobe5:perf/parse-plain-bytes

Conversation

@deadstrobe5

Copy link
Copy Markdown

Every input byte goes through the escape, quote, record delimiter, comment, delimiter, max record size and trim checks before it is appended to the current field. Most bytes of a real file are ordinary field content that can't start any of those tokens. This builds a 256-entry table once per parse call that marks the first byte of every delimiter, record delimiter, quote, escape and comment, plus CR, LF and the trim characters when ltrim or rtrim is set. A byte that isn't marked is appended directly, as long as the parser isn't escaping or inside a comment and, with rtrim, isn't right after a closing quote. Every marked byte still goes through the existing code. With max_record_size the old path is kept unchanged.

10 MB, 100k rows x 8 columns before after speedup
sync, default options 378 ms 273 ms 1.39x
sync, columns: true, trim: true 924 ms 668 ms 1.38x
stream, columns: true 637 ms 500 ms 1.27x
sync, columns: true, cast: true 822 ms 680 ms 1.21x
stream, 2 KB input x300 152 ms 154 ms 0.99x

CPU time, median of 5 to 7 runs alternating old and new code in the same process, Node 24.22.2. The machine is shared, so an A/A run of the same code varied by up to 7%. To compare outputs I ran 40k random inputs with random options (multi-byte delimiters, quotes, escapes, comments, record delimiters, trim, raw, info, BOM, utf16le and latin1, from/to, relax options, columns) through the transform API, split into random chunks down to one byte. Records, info and errors were identical. Existing tests pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant