Skip to content

perf(csv-stringify): join the sync output in batches - #562

Open
deadstrobe5 wants to merge 1 commit into
adaltas:masterfrom
deadstrobe5:perf/stringify-batch-join
Open

deadstrobe5 wants to merge 1 commit into
adaltas:masterfrom
deadstrobe5:perf/stringify-batch-join

Conversation

@deadstrobe5

Copy link
Copy Markdown

The sync API pushes every record string into one array and joins it at the end. On large datasets that keeps hundreds of thousands of small strings alive until the final join, and most of the time goes to garbage collection. The records are now joined every 1000 entries and the batches are joined at the end. The BOM and the header without records are handled as before, and Array.prototype.join still turns the BOM Buffer into the same string.

200k records x 8 fields, header: true before after speedup
sync, CPU time 1154 ms 663 ms 1.74x
sync, wall time 2479 ms 1520 ms 1.63x
sync, 20 records x5000 calls, CPU time 408 ms 391 ms 1.04x

Median of 7 runs alternating old and new code in the same process, Node 24.22.2, on a shared machine where an A/A run varied by up to 9%. A differential fuzz of 2000 random datasets (0 to 2003 records, nearly half crossing the 1000 boundary) with random options, including BOM, header, eof and casts, produced identical output. I added tests around the batch boundary and for BOM plus header without records.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant