Repository navigation
fix(csv-parse): keep a multi-character delimiter intact in raw output - #564
vanshbarfiwala-lgtm wants to merge 2 commits into
Conversation
With raw: true only the first byte of each token is copied to the raw buffer, and the delimiter branch then advanced past the rest of the delimiter without copying it. A multi-character delimiter was therefore collapsed to its first byte in record.raw (for example "a|@|b|@|c" became "a|b|c"). Copy the remaining delimiter bytes to the raw buffer. Single-character delimiters are unaffected.
| if (errField !== undefined) return errField; | ||
| if (raw === true) { | ||
| for (let i = 1; i < delimiterLength; i++) { | ||
| rawBuffer.append(buf[pos + i]); |
There was a problem hiding this comment.
The same byte loss happens for the other multi-byte tokens, which skip ahead with pos += X.length - 1 the same way, so raw is still not the original text for them:
import { parse } from "csv-parse/sync";
const raw = (s, o) => parse(s, { ...o, raw: true }).map((r) => r.raw);
raw("a,b||c,d||", { record_delimiter: "||" }); // ["a,b|", "c,d|"]
raw("%%a%%,b\n", { quote: "%%" }); // ["%a%,b\n"]
raw("a,b\r\nc,d\r\n"); // ["a,b\r", "c,d\r"] (this one is #332)Applying the same append in two more places fixes all three:
- before
const errRecord = this.__onRecord(push);(record delimiter):for (let i = 1; i < recordDelimiterLength; i++) rawBuffer.append(buf[pos + i]);(insideif (raw === true)) - before each of the three
pos += quote.length - 1;(opening/closing quote): the same loop withquote.length
With those added, all of the above round-trip exactly (plus your delimiter case, ""/\" escapes, #/// comments and trim), and the csv-parse suite still passes. Could be worth folding in here since it's the same root cause, and it would close #332 too.
There was a problem hiding this comment.
Checked 3f6b8c8: raw now matches the input for record_delimiter: "||", quote: "%%", \r\n, a two-char escape inside a two-char quote, delimiter: "::" and comment: "##", and the stream parser gives the same raw for every split point of those inputs. npm test in csv-parse passes (637). Thanks.
|
Thanks for verifying the updated fix. The PR now preserves the original raw bytes for multi-character delimiters, quotes, escapes, and CRLF endings, including the missing LF reported in #332. This prevents silent changes to CSV text used for round-tripping or auditing. The added regression tests cover sync parsing and streaming with every byte split into a separate chunk. All 637 csv-parse tests pass (two pending), along with TypeScript checks, the build, and changed-file ESLint. With these cases addressed, could a maintainer review whether this is ready to merge? I'm happy to address any remaining concerns. |
With
raw: true, the parser copies only the first byte of a token before advancing over its remaining bytes. This silently truncatesrecord.rawfor multi-character field and record delimiters, quotes, and escapes, including ordinary CRLF record endings.Copy the skipped bytes before advancing, and preserve record delimiter bytes before emitting the record. This addresses the additional cases identified in review and the missing LF reported in #332. Changes remain in
lib/; generateddist/files are unchanged.Regression tests cover field delimiters, record delimiters, CRLF, opening/closing and relaxed quotes, multi-byte escapes, and UTF-8 quotes. The new cases compare sync and streaming output, split every byte into its own stream chunk, and check parsed fields against parsing with
rawdisabled.Validation: all 637 csv-parse tests pass (two pending), including TypeScript checks; build and changed-file ESLint pass. Tested on Linux with Node 24.19.0.
AI assistance: OpenAI Codex assisted implementation and testing.