Repository navigation
fix(csv-parse): end a skipped record at its record delimiter - #568
anubhavlal07 wants to merge 2 commits into
Conversation
| } | ||
| if ( | ||
| this.state.quoting === true && | ||
| this.state.recordHasError === true |
There was a problem hiding this comment.
This also ends quoting for records whose error came from somewhere else, so a real multi-line quoted field later in a bad record gets split, and its inner lines come out as records.
import { parse } from "csv-parse/sync";
parse('id,size,note\n1,5" pipe,"first line\n2,fake,record\nlast line"\n3,real,row\n',
{ skip_records_with_error: true });
// master: [["id","size","note"],["3","real","row"]]
// this PR: [["id","size","note"],["2","fake","record"],["3","real","row"]]Here the error is INVALID_OPENING_QUOTE on 5", and "first line ... last line" is a valid quoted field. On master the whole record is skipped. With this change, 2,fake,record (text inside the quotes) is emitted as a row, and on_skip fires twice.
A fix that keeps your tests passing: only do this when the error was the invalid closing quote. Set a flag next to the CSV_INVALID_CLOSING_QUOTE error (after if (err !== undefined) return err; add this.state.quoteHasError = true;), reset it where recordHasError is reset, and check it here:
| this.state.recordHasError === true | |
| this.state.quoteHasError === true |
With that, the example above matches master, option.skip_records_with_error.ts passes (21), and the full csv-parse suite passes (633).
There was a problem hiding this comment.
Thanks, good catch. The delimiter reset now only applies when the quote state came from a skipped CSV_INVALID_CLOSING_QUOTE: a new quoteHasError flag is set there, checked in the reset block, and cleared when the record resets. I added your example as a regression test; it fails before this change and passes after, matching master's output with a single on_skip. The full csv-parse suite still passes.
There was a problem hiding this comment.
Checked the new head with that input: it now gives [["id","size","note"],["3","real","row"]], same as master. Thanks!
Fixes #567
With
skip_records_with_error: true, a record with an invalid closing quote such as2,"Bob" Jr,Initechwas skipped, but the parser stayed in quoting mode afterwards. It then read the following line breaks as part of the quoted field, so the next records were merged into the bad one and lost, or reported as extraCSV_RECORD_INCONSISTENT_FIELDS_LENGTHskips.The fix:
Records that are valid, and the existing skip behaviour (including the existing
handle "Invalid closing quote"test), are unchanged.Tests: a new
resume after "Invalid closing quote"block intest/option.skip_records_with_error.tscovers the input from the issue, an unquoted variant, CRLF line endings, and a bad first record. All four fail without the change. The full csv-parse suite passes (471 tests), andtsc --noEmitis clean.I used an AI assistant while working on this and reviewed and tested the change myself.