Skip to content

fix(csv-parse): append grouped columns in place - #571

Open
970820JH wants to merge 1 commit into
adaltas:masterfrom
970820JH:fix/group-columns-quadratic
Open

970820JH wants to merge 1 commit into
adaltas:masterfrom
970820JH:fix/group-columns-quadratic

Conversation

@970820JH

Copy link
Copy Markdown

Summary

With group_columns_by_name: true, each duplicate column is appended with obj[name] = obj[name].concat(value). That copies the whole group on every duplicate, so building one record is O(n²) in the number of duplicate headers.

Measured with csv-parse@7.0.3, sync API, one data row, every header named a:

duplicate headers input before after
10,000 40 KB 294 ms 8 ms
20,000 80 KB 1,387 ms 11 ms
40,000 160 KB 6,604 ms 29 ms

That cost repeats for every data row. When the CSV comes from an untrusted source, a small file can block the event loop for a long time.

Change

  • Append into the existing array with push. Array values are still flattened, which keeps the old concat semantics.
  • Tests:
    • grouping 3+ duplicates
    • flattening array values returned by cast (same output before and after)
    • a 50k-duplicate-column case. It runs in ~60 ms and times out without the fix.

Full csv-parse suite: 632 passing. ESLint is clean.

🤖 Generated with Claude Code

group_columns_by_name rebuilt the grouped array with concat for every
duplicate column, making each record O(n^2) in the number of duplicate
headers (40k duplicate headers, ~160 KB of input: ~6.6 s per record).
Push into the existing array instead, flattening array values to keep
concat semantics.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
// Append in place; `concat` copied the array on every
// duplicate, making grouping quadratic in the column count.
// Array values are flattened to preserve `concat` semantics.
const group = obj[columns[i].name];

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch now also runs when the first value of the column is an array that came from cast. Then group is the caller's own array, not one the parser made, so push changes it. Before, concat returned a new array, so nothing was mutated.

If cast returns the same array more than once (a shared empty value, a cache), it breaks:

const NONE = [];
parse("tag,tag\n,x\n,y\n,", {
  columns: true,
  group_columns_by_name: true,
  cast: (v, { header }) => (header ? v : v === "" ? NONE : v.split("|")),
});
// master: [{tag:["x"]},{tag:["y"]},{tag:[]}], and NONE stays []
// this PR: RangeError: Invalid array length

Rows 1 and 2 push x and y into NONE. In row 3 both cells are NONE, so group === record[i] and the for...of keeps pushing into the array it is reading from until it runs out of length.

A fix that keeps the linear cost: track which names the parser has already grouped in this record. Use concat once, when the group is created, and push after that:

const obj = {};
const grouped = new Set();
// ...
if (grouped.has(columns[i].name)) {
  // push / flatten as you have it now
} else {
  obj[columns[i].name] = Array.isArray(obj[columns[i].name])
    ? obj[columns[i].name].concat(record[i])
    : [obj[columns[i].name], record[i]];
  grouped.add(columns[i].name);
}

With that change the repro above gives the same output as master. The full csv-parse suite passes (632), and your 50k-column test still runs fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants