Skip to content

fix(csv-parse): keep a multi-character delimiter intact in raw output - #564

Open
vanshbarfiwala-lgtm wants to merge 2 commits into
adaltas:masterfrom
vanshbarfiwala-lgtm:fix/csv-parse-raw-multichar-delimiter
Open

vanshbarfiwala-lgtm wants to merge 2 commits into
adaltas:masterfrom
vanshbarfiwala-lgtm:fix/csv-parse-raw-multichar-delimiter

Conversation

@vanshbarfiwala-lgtm

@vanshbarfiwala-lgtm vanshbarfiwala-lgtm commented Nov 16, 2026 •

Copy link
Copy Markdown

With raw: true, the parser copies only the first byte of a token before advancing over its remaining bytes. This silently truncates record.raw for multi-character field and record delimiters, quotes, and escapes, including ordinary CRLF record endings.

Copy the skipped bytes before advancing, and preserve record delimiter bytes before emitting the record. This addresses the additional cases identified in review and the missing LF reported in #332. Changes remain in lib/; generated dist/ files are unchanged.

Regression tests cover field delimiters, record delimiters, CRLF, opening/closing and relaxed quotes, multi-byte escapes, and UTF-8 quotes. The new cases compare sync and streaming output, split every byte into its own stream chunk, and check parsed fields against parsing with raw disabled.

Validation: all 637 csv-parse tests pass (two pending), including TypeScript checks; build and changed-file ESLint pass. Tested on Linux with Node 24.19.0.

AI assistance: OpenAI Codex assisted implementation and testing.

With raw: true only the first byte of each token is copied to the raw
buffer, and the delimiter branch then advanced past the rest of the
delimiter without copying it. A multi-character delimiter was therefore
collapsed to its first byte in record.raw (for example "a|@|b|@|c" became
"a|b|c"). Copy the remaining delimiter bytes to the raw buffer.
Single-character delimiters are unaffected.
if (errField !== undefined) return errField;
if (raw === true) {
for (let i = 1; i < delimiterLength; i++) {
rawBuffer.append(buf[pos + i]);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same byte loss happens for the other multi-byte tokens, which skip ahead with pos += X.length - 1 the same way, so raw is still not the original text for them:

import { parse } from "csv-parse/sync";
const raw = (s, o) => parse(s, { ...o, raw: true }).map((r) => r.raw);
raw("a,b||c,d||", { record_delimiter: "||" }); // ["a,b|", "c,d|"]
raw("%%a%%,b\n", { quote: "%%" });            // ["%a%,b\n"]
raw("a,b\r\nc,d\r\n");                        // ["a,b\r", "c,d\r"]  (this one is #332)

Applying the same append in two more places fixes all three:

  • before const errRecord = this.__onRecord(push); (record delimiter): for (let i = 1; i < recordDelimiterLength; i++) rawBuffer.append(buf[pos + i]); (inside if (raw === true))
  • before each of the three pos += quote.length - 1; (opening/closing quote): the same loop with quote.length

With those added, all of the above round-trip exactly (plus your delimiter case, ""/\" escapes, #/// comments and trim), and the csv-parse suite still passes. Could be worth folding in here since it's the same root cause, and it would close #332 too.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked 3f6b8c8: raw now matches the input for record_delimiter: "||", quote: "%%", \r\n, a two-char escape inside a two-char quote, delimiter: "::" and comment: "##", and the stream parser gives the same raw for every split point of those inputs. npm test in csv-parse passes (637). Thanks.

@vanshbarfiwala-lgtm

Copy link
Copy Markdown
Author

Thanks for verifying the updated fix. The PR now preserves the original raw bytes for multi-character delimiters, quotes, escapes, and CRLF endings, including the missing LF reported in #332. This prevents silent changes to CSV text used for round-tripping or auditing.

The added regression tests cover sync parsing and streaming with every byte split into a separate chunk. All 637 csv-parse tests pass (two pending), along with TypeScript checks, the build, and changed-file ESLint.

With these cases addressed, could a maintainer review whether this is ready to merge? I'm happy to address any remaining concerns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants