Skip to content

benchmarks: prepare arguments outside the measured section - #14965

Open
lucy-muyue wants to merge 3 commits into
uutils:mainfrom
lucy-muyue:fix/benchmark-args-14903
Open

lucy-muyue wants to merge 3 commits into
uutils:mainfrom
lucy-muyue:fix/benchmark-args-14903

Conversation

@lucy-muyue

Copy link
Copy Markdown

Previously, benchmark arguments were prepared inside the measured section. This PR uses get_bench_args to prepare arguments in advance and keeps temporary-file setup, cleanup, and stdin preparation outside the measured section.

Validation: 692 unit tests passed for the affected crates and uucore. Documentation tests: 29 passed, 7 ignored. Strict Clippy remains blocked by a pre-existing warning in generated code.

Closes #14903

@codspeed

codspeed Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 1.08%

⚡ 48 improved benchmarks
❌ 8 regressed benchmarks
✅ 343 untouched benchmarks
⏩ 54 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation true_consecutive_calls 294.7 ns 34,431.5 ns -99.14%
❌ Simulation false_consecutive_calls 294.7 ns 34,149.8 ns -99.14%
❌ Simulation three_39_bit_primes 545.9 ms 870.2 ms -37.27%
❌ Simulation mv_force_overwrite 148.2 ms 191.5 ms -22.61%
❌ Simulation mv_single_file 146.8 ms 169.3 ms -13.3%
❌ Simulation sort_dictionary_order[500000] 1.9 s 2.1 s -6.54%
❌ Simulation rm_recursive_tree 13.7 ms 14.4 ms -4.96%
❌ Simulation hostname_ip_lookup[100000] 170 µs 177.5 µs -4.2%
⚡ Simulation mv_directory 8,522.1 µs 319.9 µs ×27
⚡ Memory mv_directory 224.7 KB 32.3 KB ×6.9
⚡ Memory rm_single_file 32.1 KB 20.2 KB +59.08%
⚡ Memory index_match_at_end 586.8 KB 391.4 KB +49.92%
⚡ Memory index_no_match 586.8 KB 391.4 KB +49.92%
⚡ Simulation cut_fields_custom_delim 65.9 ms 51.9 ms +27%
⚡ Memory rm_multiple_files 280.6 KB 222.1 KB +26.35%
⚡ Memory rm_force_files 281.2 KB 222.6 KB +26.31%
⚡ Simulation cut_fields_tab 57.4 ms 45.5 ms +26.24%
⚡ Simulation rm_force_files 4.6 ms 3.8 ms +19.18%
⚡ Simulation rm_multiple_files 4.9 ms 4.1 ms +18.89%
⚡ Simulation cut_bytes 18.1 ms 15.5 ms +16.69%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing lucy-muyue:fix/benchmark-args-14903 (6ba6c87) with main (8f6a838)

Open in CodSpeed

Footnotes

  1. 54 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

GNU testsuite comparison:

Skip an intermittent issue tests/date/resolution (fails in this run but passes in the 'main' branch)

@lucy-muyue

Copy link
Copy Markdown
Author

I investigated the reported regressions by comparing the baseline (84d75fcd) and this PR (ad15ddb0) on the same machine, with the same Rust toolchain and lockfile.

For true and false, inspecting the generated assembly showed that the old timed loop had been optimized into a loop producing constant return values. The updated benchmark retains argument consumption and deallocation in the measured section. The large difference therefore does not mean the utilities themselves became that much slower.

For the two mv benchmarks, local wall-clock measurements were about 10% lower. However, raw instruction counts from CodSpeed's Valgrind engine consistently increased by 6.24% and 8.09%. Most of the additional instructions in mv_force_overwrite were in a libc allocation-related path. Changes to allocation order may explain this, but that cause is not yet proven. I cannot attribute these regressions entirely to different CI machines.

For factor::hard::five_38_bit_primes, repeated measurements showed overlapping ranges and no consistent slowdown, which is consistent with variation in its randomized factorization path.

These changes affect benchmark setup and timing boundaries; the utility implementations are unchanged. The local instruction counts are not the hosted CodSpeed timing figures, and the mv simulation increases remain something to evaluate rather than dismiss.

Comment on lines +14 to +18
.with_inputs(|| {
(0..100)
.map(|_| get_bench_args(&[]).into_iter())
.collect::<Vec<_>>()
})

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.with_inputs(|| {
(0..100)
.map(|_| get_bench_args(&[]).into_iter())
.collect::<Vec<_>>()
})
.with_inputs(|| get_bench_args(&[]).into_iter())

do we need to prepare same arguments 100 times?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi! thanks for your prompt reply and sorry for my late reply, I was overwhelmed in the previous days.

For 100 times test, I place it just because the previous code is in such way:

/// Benchmark multiple consecutive invocations (avoid less than 1 ns)
#[divan::bench]
fn true_consecutive_calls(bencher: Bencher) {
bencher.bench(|| {
for _ in 0..100 {
black_box(run_util_function(uumain, &[]));
}
});
}

so I want to align with the original logic

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌ Simulation true_consecutive_calls 294.7 ns 34,431.5 ns -99.14%
❌ Simulation false_consecutive_calls 294.7 ns 34,149.8 ns -99.14%

Well this seems unexpected.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❌
Simulation
true_consecutive_calls
294.7 ns
34,431.5 ns
-99.14%

❌
Simulation
false_consecutive_calls
294.7 ns
34,149.8 ns
-99.14%

Well this seems unexpected.

My understanding is that this difference comes from compiler optimization. In the previous benchmark, argument creation and the utility call were in the same measured closure. With the fixed empty input, the compiler optimized away argument creation, consumption and deallocation, leaving a loop producing constant return values.

In the updated benchmark, arguments are prepared before measurement and passed into the measured closure as runtime inputs by the benchmark framework. In the optimized binaries we inspected, argument consumption and deallocation remain in the measured section, which explains the increased timing.

@xtqqczze

Copy link
Copy Markdown
Collaborator

Strict Clippy remains blocked by a pre-existing warning in generated code.

tell me more about this

@xtqqczze

Copy link
Copy Markdown
Collaborator

For factor::hard::five_38_bit_primes, repeated measurements showed overlapping ranges and no consistent slowdown, which is consistent with variation in its randomized factorization path.

High variance for the factor benchmarks is a known issue: #14921.

@lucy-muyue

Copy link
Copy Markdown
Author

Strict Clippy remains blocked by a pre-existing warning in generated code.

tell me more about this

for this, that's because here https://github.com/uutils/coreutils/blob/ad15ddb02310fee6307d64051e8ec42d89553b0f/src/uucore/build.rs#L25 it has:
// Generate optimized lookup function instead of HashMap writeln!(embedded_file, "#[expect(clippy::match_same_arms)]")?; // needed for dir and vdir aliases for ls writeln!( embedded_file, "pub fn get_embedded_locale(key: &str) -> Option<&'static str> {{" )?;

it is expected to have warning/error "clippy::match_same_arms", but it didn't really have, so in clippy.stderr file in my local environment, it has:
error: this lint expectation is unfulfilled --> /my_path/.cache/coreutils-14903-target/debug/build/uucore-a9cc05a7a3b914da/out/embedded_locales.rs:5:10 | 5 | #[expect(clippy::match_same_arms)] | ^^^^^^^^^^^^^^^^^^^^^^^ | = note:-D unfulfilled-lint-expectationsimplied by-D warnings= help: to override-D warningsadd#[allow(unfulfilled_lint_expectations)]`

error: could not compile uucore (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
`

@lucy-muyue

Copy link
Copy Markdown
Author

For factor::hard::five_38_bit_primes, repeated measurements showed overlapping ranges and no consistent slowdown, which is consistent with variation in its randomized factorization path.

High variance for the factor benchmarks is a known issue: #14921.

thanks for your information very much!

Preserve the new factor benchmarks with arguments prepared before measurement, and adapt the new tr benchmark to the prepared-argument helper.
@lucy-muyue
lucy-muyue requested a review from xtqqczze October 8, 2026 20:14
@xtqqczze

xtqqczze commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

error: this lint expectation is unfulfilled

I could not reproduce this locally with the following command:

cargo clippy -q --workspace --all-targets --all-features

could you please provide more information:

  • rustc --print host-tuple
  • the clippy command used where you see the error

@lucy-muyue

Copy link
Copy Markdown
Author

error: this lint expectation is unfulfilled

I could not reproduce this locally with the following command:

cargo clippy -q --workspace --all-targets --all-features

could you please provide more information:

  • rustc --print host-tuple
  • the clippy command used where you see the error

Hello! I reproduced this on Ubuntu 26.04 under WSL2, using Rust 1.98.1 and Clippy 0.1.98. rustc --print host-tuple returns x86_64-unknown-linux-gnu.

The failing command was:

export RUSTUP_TOOLCHAIN=stable
cargo clippy -q --workspace --all-targets --all-features -- -D warnings

I traced the error to my Windows checkout, which has core.symlinks=false. The base64/locales and basenc/locales symlinks were checked out as plain text files. This omitted their locale match arms, leaving #[expect(clippy::match_same_arms)] unfulfilled.

With real symlinks preserved in a Linux copy of the same commit, the same strict command passes (exit code 0). Your command without -D warnings exits successfully on the Windows checkout but emits this warning.

@xtqqczze

Copy link
Copy Markdown
Collaborator

for this, that's because here https://github.com/uutils/coreutils/blob/ad15ddb02310fee6307d64051e8ec42d89553b0f/src/uucore/build.rs#L25 it has: // Generate optimized lookup function instead of HashMap writeln!(embedded_file, "#[expect(clippy::match_same_arms)]")?; // needed for dir and vdir aliases for ls writeln!( embedded_file, "pub fn get_embedded_locale(key: &str) -> Option<&'static str> {{" )?;

Thanks, we'll address that: #15306.

@xtqqczze

Copy link
Copy Markdown
Collaborator

Let's get #15602 merged first, since that will affect benchmark results.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prepare benchmark arguments outside the measured section

2 participants