Repository navigation
benchmarks: prepare arguments outside the measured section - #14965
lucy-muyue wants to merge 3 commits into
Conversation
Merging this PR will degrade performance by 1.08%
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
|
GNU testsuite comparison: |
|
I investigated the reported regressions by comparing the baseline ( For For the two For These changes affect benchmark setup and timing boundaries; the utility implementations are unchanged. The local instruction counts are not the hosted CodSpeed timing figures, and the |
| .with_inputs(|| { | ||
| (0..100) | ||
| .map(|_| get_bench_args(&[]).into_iter()) | ||
| .collect::<Vec<_>>() | ||
| }) |
There was a problem hiding this comment.
| .with_inputs(|| { | |
| (0..100) | |
| .map(|_| get_bench_args(&[]).into_iter()) | |
| .collect::<Vec<_>>() | |
| }) | |
| .with_inputs(|| get_bench_args(&[]).into_iter()) |
do we need to prepare same arguments 100 times?
There was a problem hiding this comment.
Hi! thanks for your prompt reply and sorry for my late reply, I was overwhelmed in the previous days.
For 100 times test, I place it just because the previous code is in such way:
coreutils/src/uu/true/benches/true_bench.rs
Lines 10 to 18 in 7e94315
so I want to align with the original logic
There was a problem hiding this comment.
❌ Simulation true_consecutive_calls 294.7 ns 34,431.5 ns -99.14% ❌ Simulation false_consecutive_calls 294.7 ns 34,149.8 ns -99.14%
Well this seems unexpected.
There was a problem hiding this comment.
❌
Simulation
true_consecutive_calls
294.7 ns
34,431.5 ns
-99.14%❌
Simulation
false_consecutive_calls
294.7 ns
34,149.8 ns
-99.14%Well this seems unexpected.
My understanding is that this difference comes from compiler optimization. In the previous benchmark, argument creation and the utility call were in the same measured closure. With the fixed empty input, the compiler optimized away argument creation, consumption and deallocation, leaving a loop producing constant return values.
In the updated benchmark, arguments are prepared before measurement and passed into the measured closure as runtime inputs by the benchmark framework. In the optimized binaries we inspected, argument consumption and deallocation remain in the measured section, which explains the increased timing.
tell me more about this |
High variance for the |
for this, that's because here https://github.com/uutils/coreutils/blob/ad15ddb02310fee6307d64051e8ec42d89553b0f/src/uucore/build.rs#L25 it has: it is expected to have warning/error "clippy::match_same_arms", but it didn't really have, so in clippy.stderr file in my local environment, it has: error: could not compile |
thanks for your information very much! |
Preserve the new factor benchmarks with arguments prepared before measurement, and adapt the new tr benchmark to the prepared-argument helper.
I could not reproduce this locally with the following command: cargo clippy -q --workspace --all-targets --all-featurescould you please provide more information:
|
Hello! I reproduced this on Ubuntu 26.04 under WSL2, using Rust 1.98.1 and Clippy 0.1.98. The failing command was: export RUSTUP_TOOLCHAIN=stable
cargo clippy -q --workspace --all-targets --all-features -- -D warningsI traced the error to my Windows checkout, which has With real symlinks preserved in a Linux copy of the same commit, the same strict command passes (exit code 0). Your command without |
Thanks, we'll address that: #15306. |
|
Let's get #15602 merged first, since that will affect benchmark results. |
Previously, benchmark arguments were prepared inside the measured section. This PR uses
get_bench_argsto prepare arguments in advance and keeps temporary-file setup, cleanup, and stdin preparation outside the measured section.Validation: 692 unit tests passed for the affected crates and
uucore. Documentation tests: 29 passed, 7 ignored. Strict Clippy remains blocked by a pre-existing warning in generated code.Closes #14903