Skip to content

build(deps): upgrade codspeed-divan-compat to v6.1 - #15602

Open
xtqqczze wants to merge 1 commit into
uutils:mainfrom
xtqqczze:deps/codspeed
Open

xtqqczze wants to merge 1 commit into
uutils:mainfrom
xtqqczze:deps/codspeed

Conversation

@xtqqczze

@xtqqczze xtqqczze commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

This PR contains the following updates:

Package Type Update Change
codspeed-divan-compat (source) workspace.dependencies major 5.0.0 → 6.1.0

Release Notes

CodSpeedHQ/codspeed-rust (codspeed-divan-compat)

v6.1.0

Compare Source

Highlights
  • Add telemetry measurement mode, supported by the divan, criterion and bencher compat crates

What's Changed

Full Changelog: CodSpeedHQ/codspeed-rust@v6.0.0...v6.1.0

v6.0.0

Compare Source

CodSpeed for Rust 6.0.0 removes a source of measurement noise from your CPU Simulation benchmarks, so you can spend less time investigating unexpected changes in unrelated results.

Highlights

Your Divan and Criterion benchmarks now run in separate processes. Changes to one benchmark no longer carry allocator state into the benchmarks that follow it, improving consistency as your benchmark suite evolves.

Early results showed lower run-to-run variance, with roughly 70% fewer benchmarks varying by more than 1%.
No benchmark code changes required. Your WallTime results are unaffected.

Breaking Changes
  • Some benchmark values may shift when you upgrade
  • Each benchmark adds small process startup overhead to the run

Set CODSPEED_PROCESS_ISOLATION=false to revert to the previous behavior.

What's Changed

Full Changelog: CodSpeedHQ/codspeed-rust@v5.0.3...v6.0.0


Contributes to #14872.

@xtqqczze

This comment was marked as outdated.

@xtqqczze xtqqczze mentioned this pull request Oct 21, 2026
1 task
@codspeed

codspeed Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 62.17%

4 warnings, 1 note

⚠️ 125 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 123 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

⚠️ 75 benchmarks with a significant change ran in a different software environment

Their base and head results were measured with a different OS or toolchain, such as compiler, runtime or library versions, so the changes might come from this difference rather than from the code. Open an affected benchmark to see what changed.

See affected benchmarks

⚠️ 171 benchmarks with a significant change ran on a different major version of their CodSpeed integration

Their head results were measured with codspeed-rust 6.1.0, a different major version than their base results. A major version can change how benchmarks are measured, so the changes might come from this version difference rather than from the code. Comparisons between runs on the same major version are not affected.

See affected benchmarks

ℹ️ 1 benchmark has no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 8 improved benchmarks
❌ 163 regressed benchmarks
✅ 242 untouched benchmarks
⏩ 70 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Memory factor_multiple_u64s[2] 5.6 KB 95 KB -94.07%
❌ Memory factor_64bit_semiprime 5.7 KB 95 KB -94.03%
❌ Memory factor_large_u64_prime 5.7 KB 95 KB -94.02%
❌ CPU Simulation single_date_now 169.8 µs 2,551.7 µs -93.34%
❌ Memory single_date_now 15.9 KB 211.3 KB -92.49%
❌ Memory complex_relative_date 17.1 KB 212.3 KB -91.97%
❌ Memory b64_encode_synthetic 8.5 KB 97.8 KB -91.34%
❌ Memory file_iso_dates 19.6 KB 212.3 KB -90.74%
❌ Memory file_custom_format 20.2 KB 212.8 KB -90.53%
❌ Memory seq_integers 9.5 KB 98.8 KB -90.37%
❌ Memory seq_with_step 9.6 KB 98.9 KB -90.33%
❌ CPU Simulation complex_relative_date 260.4 µs 2,643.1 µs -90.15%
❌ Memory seq_formatted 10 KB 99.4 KB -89.89%
❌ Memory factor_three_39_bit_primes 13.6 KB 102.9 KB -86.8%
❌ Memory factor_a_prime_to_the_seventh 14.9 KB 104.7 KB -85.73%
❌ Memory factor_small_factors_and_a_large_prime 15.4 KB 104.8 KB -85.26%
❌ Memory wc_default_large_line_count[100000] 16.6 KB 105.9 KB -84.35%
❌ Memory wc_bytes_lines_synthetic[2000] 16.6 KB 105.9 KB -84.35%
❌ Memory wc_lines_large_line_count[500000] 16.6 KB 105.9 KB -84.35%
❌ Memory wc_words_synthetic[2000] 16.6 KB 105.9 KB -84.35%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing xtqqczze:deps/codspeed (0fbc81f) with main (9311390)2

Open in CodSpeed

Footnotes

  1. 70 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on main (276ffe9) during the generation of this report, so 9311390 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@xtqqczze

Copy link
Copy Markdown
Collaborator Author

@not-matthias (CodSpeed) Is the number and magnitude of reported regressions expected?

@xtqqczze
xtqqczze marked this pull request as ready for review October 21, 2026 21:40
@sylvestre

Copy link
Copy Markdown
Contributor

clippy is sad :)

@xtqqczze

This comment was marked as outdated.

@not-matthias

Copy link
Copy Markdown
Contributor

@not-matthias (CodSpeed) Is the number and magnitude of reported regressions expected?

@xtqqczze Hey yes, we're now running each benchmark in it's own process which previously caused variance issues: https://github.com/CodSpeedHQ/codspeed-rust/releases/tag/v6.0.0

Looking at the flamegraphs, it looks like you have some <std::sync::once_lock::OnceLock<jiff::tz::db::TimeZoneDatabase>>::initialize::<<std::sync::once_lock::OnceLock<jiff::tz::db::TimeZoneDatabase>>::get_or_init calls in single_date_now which was likely setup in a previous benchmark before and therefore not visible. This might be something you want to do in the setup before, rather than in the benchmark.

The same seems to be happening in the memory benchmarks, where some now have a OnceLock which was previously allocated in a separate benchmark and thus didn't show up.

@not-matthias

not-matthias commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

@xtqqczze Side note, you can also tag @codspeedbot to analyze the flamegraphs which can do the same analysis (in case I miss your ping)

@codspeedbot demonstrate for me pls: fact check me here, analyze the flamegraphs and and give concrete details on what's causing this

EDIT: ^ you can try this if you want

@xtqqczze

Copy link
Copy Markdown
Collaborator Author

@codspeedbot analyze the flamegraphs and and give concrete details on what's causing performance changes

@not-matthias

Copy link
Copy Markdown
Contributor

@codspeedbot analyze the flamegraphs and and give concrete details on what's causing performance changes

Only members of the uutils organization on CodSpeed can ask the performance wizard for help here. Once you've joined it on CodSpeed, comment again.

CC @sylvestre

@xtqqczze

xtqqczze commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Looking at the flamegraphs, it looks like you have some <std::sync::once_lock::OnceLock<jiff::tz::db::TimeZoneDatabase>>::initialize::<<std::sync::once_lock::OnceLock<jiff::tz::db::TimeZoneDatabase>>::get_or_init calls in single_date_now which was likely setup in a previous benchmark before and therefore not visible. This might be something you want to do in the setup before, rather than in the benchmark.

In this case of single_date_now, the intention is to measure "the overhead of starting the utility for a single date (no file)", so the new result is actually more accurate.

@xtqqczze

Copy link
Copy Markdown
Collaborator Author

@sylvestre once we merge this, we should likely rebase most PRs for new performance baseline :(

@xtqqczze xtqqczze mentioned this pull request Oct 28, 2026
@xtqqczze

Copy link
Copy Markdown
Collaborator Author

CI is currently red on main, so this seems like a good time to merge this PR, since all the other PRs will need to be rebased anyway.

@sylvestre
sylvestre force-pushed the deps/codspeed branch 2 times, most recently from a77be9e to dfe6aff Compare October 29, 2026 16:51
@xtqqczze

This comment was marked as outdated.

@xtqqczze
xtqqczze force-pushed the deps/codspeed branch 2 times, most recently from 3241c28 to 0e6f033 Compare October 30, 2026 12:37
@xtqqczze

This comment was marked as outdated.

@xtqqczze

xtqqczze commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator Author

@sylvestre ready

@xtqqczze

Copy link
Copy Markdown
Collaborator Author

/rerun OpenBSD

@xtqqczze
xtqqczze force-pushed the deps/codspeed branch 2 times, most recently from 7e3fdc7 to 31d1e05 Compare November 16, 2026 14:41
@xtqqczze

Copy link
Copy Markdown
Collaborator Author

@sylvestre It seems that adding benchmarks in #16439 has affected unrelated benchmarks in subsequent PRs. We need these changes to run benchmarks in separate processes to avoid such interference and ensure reliable results.

@github-actions

Copy link
Copy Markdown

Binary size comparison:

Individual binary size comparison VS main (threshold: >=5% AND >=4 KB).

Total size of compared binaries: 155.49 MB (+108 KB, +0.07%)

Significant per-binary changes:
  env     1.20 MB ->    1.28 MB  (+80 KB, +6.51%)

@xtqqczze

This comment was marked as outdated.

@github-actions

Copy link
Copy Markdown

GNU testsuite comparison:

Congrats! The gnu test tests/tail/follow-name is no longer failing!
Congrats! The gnu test tests/tail/pid-pipe is no longer failing!

@xtqqczze

Copy link
Copy Markdown
Collaborator Author

@sylvestre seems like good timing to merge now

@xtqqczze xtqqczze changed the title build(deps): upgrade codspeed-divan-compat to v6.0 build(deps): upgrade codspeed-divan-compat to v6.1 Nov 20, 2026
@xtqqczze

Copy link
Copy Markdown
Collaborator Author

Updated to v6.1.0.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants