Skip to content

fix(csv-parse): end a skipped record at its record delimiter - #568

Open
anubhavlal07 wants to merge 2 commits into
adaltas:masterfrom
anubhavlal07:fix/skip-record-at-delimiter
Open

anubhavlal07 wants to merge 2 commits into
adaltas:masterfrom
anubhavlal07:fix/skip-record-at-delimiter

Conversation

@anubhavlal07

Copy link
Copy Markdown

Fixes #567

With skip_records_with_error: true, a record with an invalid closing quote such as 2,"Bob" Jr,Initech was skipped, but the parser stayed in quoting mode afterwards. It then read the following line breaks as part of the quoted field, so the next records were merged into the bad one and lost, or reported as extra CSV_RECORD_INCONSISTENT_FIELDS_LENGTH skips.

The fix:

  • While a record already has an error and the parser is still in quoting mode, a record delimiter now ends quoting, so the bad record stops at its own line and the next record is parsed normally. If the record delimiter has not been discovered yet (the bad record is the first one), it is discovered there.
  • A record that already has an error no longer goes through the field-count check and no longer sets the expected record length, since it is about to be skipped. Without this, the bad record would also report a length error, and as the first record it would fix a wrong expected length for the rest of the file.

Records that are valid, and the existing skip behaviour (including the existing handle "Invalid closing quote" test), are unchanged.

Tests: a new resume after "Invalid closing quote" block in test/option.skip_records_with_error.ts covers the input from the issue, an unquoted variant, CRLF line endings, and a bad first record. All four fail without the change. The full csv-parse suite passes (471 tests), and tsc --noEmit is clean.

I used an AI assistant while working on this and reviewed and tested the change myself.

Comment thread packages/csv-parse/lib/api/index.js Outdated
}
if (
this.state.quoting === true &&
this.state.recordHasError === true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also ends quoting for records whose error came from somewhere else, so a real multi-line quoted field later in a bad record gets split, and its inner lines come out as records.

import { parse } from "csv-parse/sync";
parse('id,size,note\n1,5" pipe,"first line\n2,fake,record\nlast line"\n3,real,row\n',
  { skip_records_with_error: true });
// master: [["id","size","note"],["3","real","row"]]
// this PR: [["id","size","note"],["2","fake","record"],["3","real","row"]]

Here the error is INVALID_OPENING_QUOTE on 5", and "first line ... last line" is a valid quoted field. On master the whole record is skipped. With this change, 2,fake,record (text inside the quotes) is emitted as a row, and on_skip fires twice.

A fix that keeps your tests passing: only do this when the error was the invalid closing quote. Set a flag next to the CSV_INVALID_CLOSING_QUOTE error (after if (err !== undefined) return err; add this.state.quoteHasError = true;), reset it where recordHasError is reset, and check it here:

Suggested change
this.state.recordHasError === true
this.state.quoteHasError === true

With that, the example above matches master, option.skip_records_with_error.ts passes (21), and the full csv-parse suite passes (633).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good catch. The delimiter reset now only applies when the quote state came from a skipped CSV_INVALID_CLOSING_QUOTE: a new quoteHasError flag is set there, checked in the reset block, and cleared when the record resets. I added your example as a regression test; it fails before this change and passes after, matching master's output with a single on_skip. The full csv-parse suite still passes.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the new head with that input: it now gives [["id","size","note"],["3","real","row"]], same as master. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

skip_records_with_error drops valid records after an invalid closing quote

2 participants