Repository navigation
perf(csv-stringify): join the sync output in batches - #562
Open
deadstrobe5 wants to merge 1 commit into
Open
deadstrobe5 wants to merge 1 commit into
deadstrobe5 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The sync API pushes every record string into one array and joins it at the end. On large datasets that keeps hundreds of thousands of small strings alive until the final join, and most of the time goes to garbage collection. The records are now joined every 1000 entries and the batches are joined at the end. The BOM and the header without records are handled as before, and
Array.prototype.joinstill turns the BOM Buffer into the same string.header: trueMedian of 7 runs alternating old and new code in the same process, Node 24.22.2, on a shared machine where an A/A run varied by up to 9%. A differential fuzz of 2000 random datasets (0 to 2003 records, nearly half crossing the 1000 boundary) with random options, including BOM, header, eof and casts, produced identical output. I added tests around the batch boundary and for BOM plus header without records.