Skip to content

split: create empty chunks deferred by the open files limit - #16738

Open
costajohnt wants to merge 1 commit into
uutils:mainfrom
costajohnt:fix/split-n-stale-deferred-chunks
Open

costajohnt wants to merge 1 commit into
uutils:mainfrom
costajohnt:fix/split-n-stale-deferred-chunks

Conversation

@costajohnt

Copy link
Copy Markdown
Contributor

Fixes #16737

With -n N, -n l/N and -n r/N, OutFiles::init defers any output file it can't open because of the open files limit, and a deferred file is only opened once it receives data. When input ran out first, those chunks were never created, or kept their old contents when a prefix was reused, so reassembling the chunks returned bytes from the previous run.

Once input ends, create_unopened now opens and closes every chunk that was never opened, one at a time, through the existing instantiate_writer (so it still closes another writer if the limit is hit). All three -n modes go through the shared OutFiles code. With -e, the line and round robin modes skip this step, so elided chunks are still not created.

The issue's repro under ulimit -n 12 now prints ! and leaves 20 files. Before, byte mode printed !JKLMNOPQRST, and the line modes left only 9 of the 20 files.

Tests: test_limited_file_descriptors_truncates_unwritten_chunks covers 20, l/20 and r/20 under the same NOFILE limit as test_round_robin_limited_file_descriptors. It checks that a fresh directory gets all 20 empty chunks, and that reusing a prefix with a shorter input reassembles to exactly the new input. It fails on main (missing chunk files) and passes with this change. cargo test --features split test_split, clippy, cargo fmt --check and cargo doc with -Dwarnings pass locally.

One related thing I left alone: instantiate_writer treats any open error as hitting the descriptor limit, so a chunk that can't be created for another reason (for example a directory named xab) prints the "at file descriptor limit" message before the real error. That already happens on main in the round robin data path, and the reopen path in open_output_file drops the OS error, so telling the two apart needs its own change. Happy to send that as a follow-up.

With -n N, -n l/N and -n r/N, output files that could not be opened
upfront because of the open files limit were only opened once they
received data. Chunks that stayed empty were then never created, or
kept stale contents when a prefix was reused. Once input ends, open
and close every chunk that was never opened, one at a time, so all
chunks exist and are truncated like in GNU split.

Fixes uutils#16737

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

split -n leaves bytes from a previous run in chunks that should be empty

1 participant