Skip to content

ci(e2e): start Linux and tmachine jobs without waiting on macOS or package builds - #5270

Open
matthewgrossman wants to merge 3 commits into
mainfrom
ci/5268-branch-e2e-start-earlier/matthewgrossman
Open

matthewgrossman wants to merge 3 commits into
mainfrom
ci/5268-branch-e2e-start-earlier/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

Branch E2E Linux jobs were waiting on macOS builds they don't use, and the tmachine test inputs were built only after the packages finished.

Builds were one matrix per workflow, covering Linux and macOS, and a needs: dependency waits for every leg of a matrix. macOS runners often queue much longer than Linux builds take, so Linux finished early and then sat idle before E2E could start. In the 8 Branch E2E runs before this change, macOS finished after Linux in 5, holding Linux consumers back by 4 to 14 minutes (mean about 8.5).

This PR makes each platform its own workflow, so Linux consumers depend on Linux builds and macOS jobs depend on macOS builds. Adding Windows later means adding a workflow, not another flag or matrix branch. It also builds the tmachine test inputs at the start of the run. In an earlier Branch E2E run of this PR, this saved about 7 minutes on the critical path: about 4 from not waiting for macOS and about 3 from building the test inputs early. The macOS part grows when macOS runners queue. Timings are under Testing.

Related Issue

Closes #5268

Changes

  • build-binaries.yml and build-vm-driver.yml are replaced by per-platform workflows: build-binaries-linux.yml, build-binaries-macos.yml, build-vm-driver-linux.yml and build-vm-driver-macos.yml. Each has a plain matrix over its platform's targets, with no platform flags or conditionals. Job names and artifact names are unchanged.
  • The steps the two VM driver workflows share (downloading the sandbox, supervisor and VM init artifacts, building the runtime, building the driver) live in .github/actions/build-vm-driver.
  • Branch E2E calls each platform separately. build-binaries-linux and build-vm-driver-linux replace build-binaries and build-vm-driver as the dependencies of the Linux host lanes. build-binaries-macos and build-vm-driver-macos build macOS, and the macOS VM driver also needs the Linux musl binaries.
  • Core E2E result depends on both macOS jobs. GPU E2E result and the Kubernetes result jobs depend on build-binaries-macos. A macOS build failure still fails them, as it does today.
  • The macOS matrix uses fail-fast: false. The four darwin builds used to be separate jobs, so one failure didn't cancel the others, and this keeps it that way.
  • New Branch E2E job Build tmachine test inputs needs only PR metadata. It runs the same three Nix builds as prepare (test archives, test workload images, Helm chart) on a GitHub-hosted runner and uploads openshell-tmachine-test-inputs.
  • prepare-integration-inputs.yml takes test-inputs-artifact-name. When it's set, prepare skips Nix setup and downloads the inputs. Otherwise it runs its original build steps unchanged. The build steps stay inline rather than in a local composite action because prepare checks out the candidate commit, and older candidates (for example, a re-qualified prerelease tag) wouldn't contain that action. The openshell-tmachine-inputs layout is unchanged.
  • Release Dev and Release now call both platform workflows. Every job that depended on build-binaries or build-vm-driver depends on both platforms, so release gating is unchanged. Release Qualification, Pre-release and Integration Tests are unchanged. ci: start Release Dev Linux jobs without waiting on macOS builds #5269 tracks applying the Linux-first dependency changes to Release Dev.

Testing

Checked items and timings ran against the earlier version of this PR (platform flags on the shared workflows), which has the same Branch E2E dependency graph. Unchecked items cover the per-platform split.

  • actionlint: no new findings (the 8 existing concurrency.queue findings are in untouched files)
  • zizmor (--persona=regular --min-severity=high): same 13 findings as main
  • mise run markdown:lint
  • Branch E2E Checks pass on the earlier version of this PR (run 41894485630: 87 jobs succeeded, none failed, no reruns)
  • Branch E2E Checks pass on the per-platform split (run 41920547094, pending)
  • Release Dev still builds both platforms and gates correctly (not dispatched from this branch; it publishes from main)

Timings

Times are minutes from the start of the run. Runner load differs between runs, so the fairest comparison is this PR's run against the same run replayed through the old dependency graph. The replay keeps each job's measured queue time and duration and changes only what it waits for.

Milestone Before: 41752143964 Before, macOS queued: 41748856075 This PR, old graph (replay) This PR: 41894485630
Linux binaries done 8:13 7:02 9:29 9:29
macOS binaries done 6:37 19:28 13:50 13:50
DEB/RPM packages done 19:26 35:05 ~25:47 21:27
Prepare duration 3:55 5:30 ~4:08 1:21
First tmachine row starts 23:29 40:42 ~30:51 23:44
Core E2E result 42:21 57:09 ~47:51 40:44
  • Not waiting for macOS moved the package builds 4:20 earlier in this run, which is how long macOS binaries took after Linux binaries finished. In 41748856075 that gap was 12:26.
  • Prepare took 1:21 instead of about 4:08. It downloaded the test inputs in 31 seconds instead of spending 3:18 on Nix setup and builds. Build tmachine test inputs finished at 3:53, long before the packages.
  • The host lanes start about 4 minutes earlier and still finish well before the tmachine rows (last host lane 32:22, last tmachine row 40:34).
  • The replay assumes job durations and queue times don't depend on start time. Both "before" runs used an older version of the workflow and are for scale only.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@matthewgrossman matthewgrossman added test:e2e Requires end-to-end coverage area:build Related to CI/CD and builds topic:testing labels Nov 4, 2026
@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown

Label test:e2e applied for 3b4bd97. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Branch E2E built the macOS binaries in the same reusable workflow call
as the Linux binaries, so every Linux consumer waited for the macOS
runners. Add linux and macos inputs to build-binaries and
build-vm-driver, and call them separately in Branch E2E. The result
jobs still depend on the macOS builds, so a macOS build failure still
fails them. Release workflows keep both inputs at their defaults.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Prepare tmachine inputs built the test archives, test workload images,
and Helm chart after the DEB and RPM builds finished, which added about
three minutes before any tmachine suite could start. None of them use
the packages. Build them in a Branch E2E job that needs only PR
metadata, and let prepare download the result through a new
test-inputs-artifact-name input. Other callers of
prepare-integration-inputs leave the input empty and still build in
place with the same steps.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman
matthewgrossman force-pushed the ci/5268-branch-e2e-start-earlier/matthewgrossman branch from 3b4bd97 to 1aa0327 Compare November 4, 2026 19:11
Branch E2E used flags on the shared build workflows to keep Linux and
macOS apart. Make each platform its own reusable workflow instead, with
a plain matrix, so a consumer depends only on the platforms it needs and
adding a platform means adding a workflow rather than another flag.

Move the VM driver build steps shared by both platforms into a
composite action. Release workflows call both platforms and keep their
existing gating.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:build Related to CI/CD and builds test:e2e Requires end-to-end coverage topic:testing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: start Branch E2E Linux and tmachine jobs without waiting on macOS builds or package builds

1 participant