Skip to content

csplit: use VecDeque instead of Vec for FIFO buffer - #10925

Open
devnexen wants to merge 3 commits into
uutils:mainfrom
devnexen:csplit_vecdeque_optim
Open

devnexen wants to merge 3 commits into
uutils:mainfrom
devnexen:csplit_vecdeque_optim

Conversation

@devnexen

Copy link
Copy Markdown
Contributor

Replace Vec with VecDeque in InputSplitter to avoid O(n) shifts on remove(0). Use push_front/pop_front/push_back for O(1) queue operations.

@github-actions

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/shuf/shuf-reservoir (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/sort/sort-stale-thread-mem (fails in this run but passes in the 'main' branch)
Note: The gnu test tests/cp/link-heap is now being skipped but was previously passing.
Note: The gnu test tests/rm/many-dir-entries-vs-OOM is now being skipped but was previously passing.
Congrats! The gnu test tests/expand/bounded-memory is now passing!

@sylvestre

Copy link
Copy Markdown
Contributor

please provide hyperfine output, merci!

@devnexen

Copy link
Copy Markdown
Contributor Author

however this one does not seem worth it. main wins.

@devnexen devnexen closed this Feb 14, 2026
@devnexen devnexen reopened this Feb 14, 2026
@devnexen

Copy link
Copy Markdown
Contributor Author
hyperfine   --warmup 3   --prepare 'rm -f xx*'   -L csplit /usr/bin/csplit,"./target/profiling/coreutils.main csplit","./target/profiling/coreutils.gh10925 csplit"   "{csplit} somefile.txt 100 {9999} -z"
Benchmark 1: /usr/bin/csplit somefile.txt 100 {9999} -z
  Time (mean ± σ):     110.0 ms ±   2.0 ms    [User: 43.5 ms, System: 66.2 ms]
  Range (min … max):   106.8 ms … 113.2 ms    18 runs
 
Benchmark 2: ./target/profiling/coreutils.main csplit somefile.txt 100 {9999} -z
  Time (mean ± σ):     100.9 ms ±   4.0 ms    [User: 35.0 ms, System: 65.3 ms]
  Range (min … max):    95.6 ms … 110.1 ms    20 runs
 
Benchmark 3: ./target/profiling/coreutils.gh10925 csplit somefile.txt 100 {9999} -z
  Time (mean ± σ):      98.8 ms ±   2.5 ms    [User: 34.3 ms, System: 64.2 ms]
  Range (min … max):    94.2 ms … 104.0 ms    21 runs
 
Summary
  ./target/profiling/coreutils.gh10925 csplit somefile.txt 100 {9999} -z ran
    1.02 ± 0.05 times faster than ./target/profiling/coreutils.main csplit somefile.txt 100 {9999} -z
    1.11 ± 0.04 times faster than /usr/bin/csplit somefile.txt 100 {9999} -z
seq 1 1000000 > somefile.txt

@github-actions

Copy link
Copy Markdown

GNU testsuite comparison:

GNU test failed: tests/pr/bounded-memory. tests/pr/bounded-memory is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/shuf/shuf-reservoir (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/sort/sort-stale-thread-mem (fails in this run but passes in the 'main' branch)
Skip an intermittent issue tests/tail/symlink (fails in this run but passes in the 'main' branch)
Congrats! The gnu test tests/expand/bounded-memory is now passing!

@sylvestre

Copy link
Copy Markdown
Contributor

as we don't have benchmarks, here is one: #10927

@sylvestre
sylvestre force-pushed the csplit_vecdeque_optim branch from dce735f to 9f1afa3 Compare February 14, 2026 13:06
@github-actions

Copy link
Copy Markdown

GNU testsuite comparison:

GNU test failed: tests/pr/bounded-memory. tests/pr/bounded-memory is passing on 'main'. Maybe you have to rebase?
GNU test failed: tests/tail/retry. tests/tail/retry is passing on 'main'. Maybe you have to rebase?
Skip an intermittent issue tests/tty/tty-eof (fails in this run but passes in the 'main' branch)
Skipping an intermittent issue tests/tail/follow-name (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/factor/t36 is no longer failing!
Note: The gnu test tests/dd/no-allocate is now being skipped but was previously passing.
Note: The gnu test tests/expand/bounded-memory is now being skipped but was previously passing.
Congrats! The gnu test tests/printf/printf-surprise is now passing!

@ChrisDryden

Copy link
Copy Markdown
Collaborator

It does not appear to have detected a performance difference with the benchmarks, can you confirm if the benchmarks applied to the use case here?

@devnexen

Copy link
Copy Markdown
Contributor Author

The difference is pretty small locally after several runs more or less what seen here.

@sylvestre

Copy link
Copy Markdown
Contributor

hi, this one can't be merged as it is:

  • failing: Aggregate GNU test results, Build (ubuntu-latest, x86_64-unknown-redox, feat_os_unix_redox, redoxer, true), Separate Builds (individual and coreutils) (macos-latest, feat_os_macos)
  • it needs a rebase
  • src/uu/csplit/src/csplit.rs: could be if let Some(line) = self.buffer.pop_front() { return Some(line); } to avoid the unwrap, no?
  • src/uu/csplit/src/csplit.rs: please avoid adding a new unwrap here, if let Some(...) would work
    could you please have a look?

@devnexen
devnexen force-pushed the csplit_vecdeque_optim branch from 9f1afa3 to aa3f048 Compare October 23, 2026 08:54
Copilot AI balanced review requested due to automatic review settings October 23, 2026 08:54
@devnexen

Copy link
Copy Markdown
Contributor Author

sorry for late reply, the gain shows with a negative offset ...

seq 1 2000000 > in.txt
hyperfine -N --warmup 2 -L off 1,1000,10000 -L v main,gh10925 "./csplit.{v} -s in.txt /^2000000$/-{off}"

offset main gh10925
-1 77 ms 86 ms
-1000 433 ms 88 ms
-10000 10.5 s 87 ms
-100000 2.8 min 168 ms

@sylvestre

Copy link
Copy Markdown
Contributor

let's see if codspeed says anything here :)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

1 open issue
What changed in this PR

This PR updates InputSplitter’s internal FIFO buffer to use VecDeque instead of Vec, avoiding O(n) shifts when removing from the front.

Changes:

  • Replace Vec with VecDeque for the internal buffer.
  • Switch front operations from insert(0)/remove(0) to push_front/pop_front.
  • Use push_back for enqueueing new items in FIFO order.
File Description
src/​uu/​csplit/​src/​csplit.rs Migrates InputSplitter buffering to VecDeque to make front removals O(1).

Review effort: Lite

Comment thread src/uu/csplit/src/csplit.rs Outdated
Comment on lines +607 to +610
self.buffer.push_back((ln, Ok(line)));
self.buffer
.pop_front()
.map(|(_, head_line)| head_line.unwrap())
Copilot AI balanced review requested due to automatic review settings October 23, 2026 09:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues

Review effort: Lite

Comment thread src/uu/csplit/src/csplit.rs Resolved
@github-actions

Copy link
Copy Markdown

Binary size comparison:

Individual binary size comparison VS main (threshold: >=5% AND >=4 KB).

Total size of compared binaries: 155.05 MB (-300 KB, -0.19%)

Significant per-binary changes:
  date     2.61 MB ->    2.30 MB  (-316 KB, -11.81%)

@sylvestre

Copy link
Copy Markdown
Contributor

thanks for the work on this!
the unwrap() in add_line_to_buffer is still there, please replace it with an if let like suggested earlier.
it also needs a rebase on top of main, thanks

Replace Vec with VecDeque in InputSplitter to avoid O(n) shifts
on remove(0). Use push_front/pop_front/push_back for O(1)
queue operations.
@devnexen
devnexen force-pushed the csplit_vecdeque_optim branch from 99a5698 to 6aa1dcd Compare November 20, 2026 16:56
Copilot AI balanced review requested due to automatic review settings November 20, 2026 16:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open issues
1 resolved since last review

Review effort: Lite

Comment on lines +608 to +615
// only `Ok` lines are ever stored in the buffer
let head_line = if let Some((_, Ok(head_line))) = self.buffer.pop_front() {
Some(head_line)
} else {
None
};
self.buffer.push_back((ln, Ok(line)));
head_line

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants