Skip to content

ci: plan builds and artifact reuse per bindings root - #3321

Open
rwgk wants to merge 3 commits into
rwgk/cb12-staged-05-releasesfrom
rwgk/cb12-staged-06-planning
Open

rwgk wants to merge 3 commits into
rwgk/cb12-staged-05-releasesfrom
rwgk/cb12-staged-06-planning

Conversation

@rwgk

@rwgk rwgk commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

— codex GPT-6 on behalf of Ralf —

Layer 6 of 7 of the staged "CUDA 12 bindings on main" stack. Previous: #3319. Next: #3322. Stack map at the bottom.

What this PR does

Teaches CI to do only the work a change needs, per bindings root. Before this layer every CI run built and tested both majors in full.

  • ci/tools/compute_ci_plan.py: planning targets become (bindings, root), (core, cuNN), (python, root), and pathfinder. A cuda_bindings_12/ edit rebuilds the CUDA 12 bindings and metapackage, reuses Core, Pathfinder, and CUDA 13 wheels from the baseline run at the exact merge base, and tests only CUDA 12 rows. Core edits rebuild both ABI halves. Shared or unknown paths build everything. A bare v12.* or v13.* tag builds only its own line and uses published Pathfinder.
  • ci/tools/select_test_matrix.py (new, 113 lines) replaces the jq program that was duplicated in both test-wheel workflows: per-row suite selection, local versus published floor bindings, runner-driver mapping, and a guard so a PR cannot silently lose a required local-wheel row. Job names now say what runs, for example pathfinder+bindings12+core12+meta12 / py3.12, CTK12.9.1 (wheels; bindings PR).
  • build-wheel.yml: one jq step turns the workplan into BUILD_BINDINGS_CURRENT, TEST_CORE_MAINTENANCE, and friends; every step is gated per root; reusable artifacts are downloaded from the baseline and re-uploaded under the new SHA. Inputs cuda-version / prev-cuda-version are replaced by bindings-config.
  • ci.yml: ci-vars is folded into detect-changes, which now emits bindings-config and workplan; the Windows ARM64 build is gated by the plan.
  • Test workflows check out ci/tools from the workflow revision (.ci-control), so nightly can test an older artifact source with current tooling.
  • sdist jobs and the pixi source-build smoke select roots from the plan.

After this merges

  • PR CI work drops for single-root changes; pushes to main, scheduled runs, and non-bindings tags still build and test everything.
  • Artifact reuse engages only when the base branch has a successful push run at the exact merge base with the full artifact set. On main that becomes true after the first post-merge push run; before that, PRs build everything.
  • Scheduled GPU source tests cover both registered environments; manual dispatch selects one and defaults to the current environment.

Where to look

All changes (17 files, +1,649 / -946)

Workflows: build-wheel.yml, ci.yml, ci-pixi-source-test.yml (plan-driven root selection, docs build for changed docs/), test-sdist-linux.yml, test-sdist-windows.yml, test-wheel-linux.yml, test-wheel-windows.yml.

Tools: compute_ci_plan.py (per-root targets, release tags, GitHub output writer), select_test_matrix.py (new), bindings_config.py (write-build-env), env-vars (Python-written build variables; registry or legacy test routing), run-tests (published Pathfinder for focused releases).

Tests: test_compute_ci_plan.py (rewritten around outcomes), test_select_test_matrix.py (new), test_merge_cuda_core_wheels.py (new), ci/tools/tests/AGENTS.md (new: keep a test only when a wrong result would be silent, per Mike's review on #3176).

Validation
Stack map: what main can do after each layer merges
Layer PR After it merges, main ...
1 #3314 behaves exactly as before; CI has one reader for the two bindings roots; CUDA 13 versions can no longer match a v12.* tag
2 #3315 lets a developer build, test, and document cuda_bindings_12/; the PR-time CPU source-build smoke covers both roots; three CUDA 12 runtime fixes are in
3 #3317 builds the CUDA 12 bindings wheel itself and runs its GPU tests; cuda.core and the metapackage still consume 12.9.x artifacts
4 #3318 uses same-run CUDA 12 wheels everywhere; PR CI no longer reads the 12.9.x branch
5 #3319 can release CUDA 12 from a v12.9.N tag on main; CUDA 13 releases are unchanged
6 #3321 builds and tests only what a change needs, per bindings root, reusing baseline artifacts
7 #3322 carries the schema-2 registry, a manual-only backport workflow, a metadata consistency hook, and the maintenance docs; 12.9.x is frozen

Every layer was designed to be mergeable on its own and had a green full CI run at its pre-rebase head on 2026-10-31. The stack is the "individually mergeable pieces" alternative to #3174-#3182; both stacks reach the same end state except that this one keeps compatibility routes for older artifact sources and retained CUDA 13 tags.

@copy-pr-bot

copy-pr-bot Bot commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@rwgk
rwgk added this pull request to stack #3316 October 31, 2026 05:45
@github-actions github-actions Bot added the CI/CD CI/CD infrastructure label Oct 31, 2026
@rwgk rwgk added this to the cuda.bindings next milestone Oct 31, 2026
@rwgk rwgk added enhancement Any code-related improvements cuda.bindings Everything related to the cuda.bindings module labels Oct 31, 2026
@rwgk rwgk self-assigned this Oct 31, 2026
@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch 2 times, most recently from fdc9d18 to ba05a1d Compare October 31, 2026 07:49
@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch from ba05a1d to 463bd2c Compare October 31, 2026 08:03

rwgk commented Oct 31, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 463bd2c

@github-actions

Copy link
Copy Markdown
Contributor

rwgk commented Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 88a2ab3

@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch 4 times, most recently from 7418aad to c1b95b6 Compare November 6, 2026 21:38
rwgk added 3 commits November 7, 2026 09:44
Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
Signed-off-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
@rwgk
rwgk force-pushed the rwgk/cb12-staged-06-planning branch from c1b95b6 to 1272072 Compare November 7, 2026 17:49
@copy-pr-bot

copy-pr-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@rwgk
rwgk marked this pull request as ready for review November 8, 2026 07:14
@coderabbitai

coderabbitai Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Explain Change →

Summary

Summary by CodeRabbit

  • Release Process
    • Tagged releases now build and validate artifacts for the CUDA line associated with the tag, rather than building artifacts for multiple lines.
    • Release builds validate against the published Pathfinder dependency.
  • CI Improvements
    • Build and test coverage now follows the affected packages and supported CUDA variants, with reusable artifacts used when available.
    • CI checks release artifact coverage across supported platforms and reports selected test coverage.
  • Documentation
    • Updated the release guide to reflect the tagged-release build process.

CI now derives build and test targets from the bindings registry and workplan. Workflows use package-root and CUDA-variant decisions to select artifacts, dependencies, test suites, release builds, and source smoke checks.

Changes: Registry-driven CI

Layer / File(s) Summary
Configuration and workplan
ci/tools/bindings_config.py, ci/tools/compute_ci_plan.py, ci/tools/tests/test_compute_ci_plan.py, ci/tools/tests/test_bindings_config.py
The configuration exports package metadata and build environment values. The planner selects targets from configured package roots, changed paths, baselines, and release tags. Tests cover target selection, linked paths, and release cases.
CI orchestration
.github/workflows/ci.yml
detect-changes provides registry and platform data to downstream jobs. Workplan generation and reusable-artifact checks use registered CUDA versions, while build, sdist, and test jobs consume the plan.
Build and artifact routing
.github/workflows/build-wheel.yml, .github/workflows/test-sdist-*, .github/RELEASE-bindings.md, ci/tools/tests/test_merge_cuda_core_wheels.py
Wheel builds and reuse use current and maintenance CUDA variants, with Pathfinder selected from an artifact or the published package. Sdist jobs use package-root and variant decisions. Release documentation describes tag-selected artifacts; a test checks merged CUDA Core wheel contents.
Test matrix and dependency routing
ci/tools/select_test_matrix.py, ci/tools/env-vars, ci/tools/run-tests, ci/tools/tests/test_select_test_matrix.py, ci/tools/tests/AGENTS.md, .github/workflows/test-wheel-*
Matrix selection assigns suites and package sources. Test workflows route artifact downloads and installation from those selections. Environment and test scripts validate artifact sources and wheel counts.
Source smoke and ABI round trip
.github/workflows/ci-pixi-source-test.yml
The workflow selects packages dynamically, builds and checks source imports and Cython extensions, tests an ABI round trip, and runs GPU tests for the selected packages.

Possibly related issues

Possibly related PRs

  • NVIDIA/cuda-python#3176 — It changes the same matrix-selection and wheel-test interfaces, including bindings-config and select_test_matrix.
  • NVIDIA/cuda-python#3322 — It changes the bindings registry and release documentation, which overlap with the registry and tag-release handling here.

Suggested reviewers: mdboom, leofang, jpascucci-nv

Change: Other

Merge Risk: Low · up to 12720

A docs-only or ignored-file change under a bindings test directory can fail the source-smoke check even though no real build work is needed. This only affects CI noise on narrow PRs, and a fallback selection fixes it. Otherwise the planning change looks sound.

Security architecture risk: Low · up to 12720

The redesign retains exact-commit reuse checks, falls back to full builds when a usable baseline is unavailable, and adds protection against silently dropping required local-wheel tests. No introduced security vulnerability was established, but artifact-content validation and some execution paths remain only partially assessed.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • observed — The affected execution scope includes wheel outputs across the registered CUDA lines and matrix-selected GPU runners. Linux custom-driver rows can use privileged containers with host PID access, making matrix and driver controls security-sensitive.

Trust Boundaries and Controls

  • observed — Reuse is tied to the repository's base branch, push event, ci.yml workflow, successful status, and exact merge-base commit. The returned head SHA is checked again before the baseline run ID is accepted, preventing substitution with a merely newer base-branch run.
  • observed — Both wheel-test workflows separate the requested test-source checkout from a .ci-control checkout at github.sha, with credential persistence disabled for the latter. Configured matrix selection uses the .ci-control helper. This separates artifact-source revision from selected control helpers, but github.sha remains the caller's workflow revision rather than an independently protected revision.

Resilience and Maintainability Implications

  • observed — After caller filtering, standard pull-request selection rejects an empty matrix or omission of an enabled package root requiring local bindings or metapackage tests. Nightly and nonstandard modes intentionally do not receive the same local-wheel coverage guard; empty-registry callers retain the legacy route.

Hardening Proposals

  • proposed — Consider validating expected wheel identity, compatibility tags, and cardinality after baseline download and before republication. This would make payload assumptions explicit alongside the existing run-identity and artifact-name checks; it is not an established vulnerability.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Additional comments: 1

.github/workflows/ci-pixi-source-test.yml: 1

241-241: Trivial · Stability & Availability · Quick win

Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.

suggestion: Read the package loop from a dedicated file descriptor, not from stdin.

done < <(jq -c '.[]' ...) makes the process substitution the stdin of every command in the loop body. pixi run ... and the tools it starts inherit this stdin. If any of them reads stdin, it consumes the remaining package lines. The loop then ends early without an error, and later packages are not source-built or tested. This failure is silent, and with -e nothing flags it. The same pattern applies to the docs loop that ends at Line 250.

Use read -u 3 with 3< so that child processes cannot reach the package list.

-          while IFS= read -r bindings_package; do
+          while IFS= read -r -u 3 bindings_package; do
 ...
-          done < <(jq -c '.[]' <<< "${BINDINGS_PACKAGES}")
+          done 3< <(jq -c '.[]' <<< "${BINDINGS_PACKAGES}")

Make the same change in the docs loop.

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between d966cca and 1272072.

Files selected for processing: 18

  • .github/RELEASE-bindings.md
  • .github/workflows/build-wheel.yml
  • .github/workflows/ci-pixi-source-test.yml
  • .github/workflows/ci.yml
  • .github/workflows/test-sdist-linux.yml
  • .github/workflows/test-sdist-windows.yml
  • .github/workflows/test-wheel-linux.yml
  • .github/workflows/test-wheel-windows.yml
  • ci/tools/bindings_config.py
  • ci/tools/compute_ci_plan.py
  • ci/tools/env-vars
  • ci/tools/run-tests
  • ci/tools/select_test_matrix.py
  • ci/tools/tests/AGENTS.md
  • ci/tools/tests/test_bindings_config.py
  • ci/tools/tests/test_compute_ci_plan.py
  • ci/tools/tests/test_merge_cuda_core_wheels.py
  • ci/tools/tests/test_select_test_matrix.py
Files with no reviewable changes: 1

  • ci/tools/tests/test_bindings_config.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Comment on lines +197 to +200
if [[ "${selected_packages}" == '[]' ]]; then
echo "::error::no bindings package selected for source-build smoke tests"
exit 1
fi

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Functional Correctness · Quick win

Supported by static analysis

Script executed:

#!/bin/bash
# Inspect how compute_ci_plan classifies pixi/workflow paths and validates --baseline-run-id.
rg -nP -C4 'baseline[-_]run[-_]id' ci/tools/compute_ci_plan.py
rg -nP -C3 'setup-pixi|pixi-version|ci-pixi-source-test|\.github/' ci/tools/compute_ci_plan.py
rg -nP -C3 'force_all' ci/tools/compute_ci_plan.py

Repository: NVIDIA/cuda-python

Length of output: 5917


Script executed:

#!/bin/bash
git rev-parse HEAD
git diff --stat d966cca934a9ff7e2c50f2295f2c490451ba0195 1272072927a4dc32be7e18e9e98ad10d122600e0 -- .github/workflows/ci-pixi-source-test.yml ci/tools/compute_ci_plan.py
nl -ba .github/workflows/ci-pixi-source-test.yml | sed -n '1,245p'
nl -ba ci/tools/compute_ci_plan.py | sed -n '1,280p'
nl -ba ci/tools/compute_ci_plan.py | sed -n '300,355p'

Repository: NVIDIA/cuda-python

Length of output: 29189


Script executed:

#!/bin/bash
git ls-files '*pyproject.toml' 'cuda_bindings*/cuda/bindings/*.md' 'cuda_bindings*/cuda/bindings/*/*.md' 'cuda_core/docs/*' 'cuda_bindings*/tests/cython/*.md' 'cuda_core/tests/cython/*.md'
nl -ba ci/tools/compute_ci_plan.py | sed -n '280,305p'
nl -ba ci/tools/bindings_config.py | sed -n '1,160p'

Repository: NVIDIA/cuda-python

Length of output: 11029


Script executed:

#!/bin/bash
python3 - <<'PY'
import subprocess
paths = subprocess.check_output(["git", "ls-files", "-z"]).decode().split("\0")
for p in paths:
    parts = p.split("/")
    if (len(parts) > 3 and parts[0].startswith("cuda_bindings") and parts[1:3] == ["cuda", "bindings"] and (p.endswith((".md", ".svg")) or parts[-1] in {"AGENTS.md", "CLAUDE.md"})):
        print(p)
    if (len(parts) > 2 and (parts[0].startswith("cuda_bindings") or parts[0] == "cuda_core") and parts[1:3] == ["tests", "cython"] and parts[-1] in {"AGENTS.md", "CLAUDE.md"}):
        print(p)
PY

Repository: NVIDIA/cuda-python

Length of output: 156


important: Select packages when a triggered PR has an empty plan.

A PR that only adds cuda_bindings_12/tests/cython/AGENTS.md can trigger this workflow, but the planner ignores that basename. No fallback selects a package, so the smoke job fails. The workflow and setup-pixi paths do force all targets.

Fallback for pull requests

             done < <(git diff --no-renames --name-only -z "${merge_base}" HEAD)
+            if [[ "${selected_packages}" == '[]' ]]; then
+              selected_packages="${bindings_packages}"
+            fi
           fi
Tools

zizmor (1.31.1)

[warning] 25-421: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block

(excessive-permissions)


[warning] 79-264: overly broad permissions (excessive-permissions): default permissions used due to no permissions: block

(excessive-permissions)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI/CD CI/CD infrastructure cuda.bindings Everything related to the cuda.bindings module enhancement Any code-related improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant