Skip to content

ci: average shard timings with the previous run's - #188

Closed
HMarzban wants to merge 2 commits into
ci/four-shards-perf-split-parallel-unitsfrom
ci/smooth-shard-timings
Closed

HMarzban wants to merge 2 commits into
ci/four-shards-perf-split-parallel-unitsfrom
ci/smooth-shard-timings

Conversation

@HMarzban

Copy link
Copy Markdown
Contributor

Summary

Shard timings are averaged across runs. This follow-up is stacked on #186 (base: its branch) and is meant to land with it.

One run's per-file times are noisy. In #186's run, mobile-bar.spec.ts took 263 s against 116 s in the run its timings came from, so shard 1 carried 1,017 s of tests against 879 s planned and was the slowest job, at 7m23s.

  • smoothTimings in scripts/e2e-shards.ts is a pure exponential moving average: half weight to the newest run, half to the previous average. A new spec keeps its own time, and a deleted spec is dropped.
  • The publish step ("Write shard timings", which runs only on green pushes to main) passes the timings this run packed by (needs.changes.outputs.timings) as the previous ones. No second download is needed, and a PR still never writes timings. Without usable previous timings it writes the run's own, as before.
  • Tests:
    • e2e-shards.test.ts covers the average: a table of cases plus the weight.
    • ci-workflow.test.ts pins that the publish step reads the timings the run packed by.
  • Docs: TESTING and CHANGELOG.

Verification

  • bun test tests/unit/ci passes 176 tests, and actionlint is clean.
  • On a real merged report: mobile-bar at 130.6 s averaged with a previous 50 s gives 90.3 s. A file missing from the report is dropped. An empty previous file writes the run's own timings, byte for byte the same as without a previous file.
  • In CI: the smoothing only runs when a green push to main publishes, so it first takes effect after this merges.

Deploy: none.

🤖 Generated with Claude Code

When a green push to main publishes e2e-timings, each file's time is
averaged half and half with the timings that run packed by, so one noisy
run moves the plan only halfway. New specs keep their own time, deleted
ones drop out, and without usable previous timings the run's own are
written.
@HMarzban
HMarzban marked this pull request as ready for review November 22, 2026 15:40
@HMarzban

Copy link
Copy Markdown
Contributor Author

Included in integrate batch 12 (#190): its head e20794d is on main at 44d48d8. Closing since it was stacked on #186's branch.

@HMarzban HMarzban closed this Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant