CI: Fix compute-sanitizer setup and harden mini-CTK fetching by juenglin · Pull Request #3397 · NVIDIA/cuda-python

juenglin · 2026-11-05T23:14:47Z

Description

Follow-up to #3347. This PR fixes the compute-sanitizer setup and hardens and simplifies how CI fetches the mini CTK.

Bug fix: CTK 13.4 prerelease integration #2437 unintentionally broke sanitizer mode. It made fetch_ctk empty cuda-path before every install, so the sanitizer step, which runs fetch_ctk a second time for cuda_sanitizer_api alone, deleted the mini CTK installed just before it. The instrumented test runs then failed at startup because the toolkit headers were gone, and CTK 13.4 prerelease integration #2437 disabled the only matrix rows that run the sanitizer. fetch_ctk gains an install-mode: add input that installs on top of the existing mini CTK and keeps the environment of the first call; the sanitizer step uses it. DO NOT MERGE: probe compute-sanitizer on Python 3.12 local-CTK rows #3400 exercises the fix on Python 3.12 local-CTK rows.
Hardening: fetch_ctk now fails closed when a public-server CUDA version has no pinned manifest hash, instead of warning and continuing. The per-archive hash checks from ci: harden fetch_ctk with manifest/archive verification and optional redist-base-url #3347 are only as trustworthy as that manifest, and since main moved to 13.5.0 it has gone unverified. This PR pins 13.5.0 and adds a test that requires a pin for every CUDA version in ci/versions.yml and every LOCAL_CTK: '1' row of ci/test-matrix.yml, so the next bump fails before merge. Manifests from an alternate redist server stay warn-only, since the public pins do not apply to them.
Simplification: every fetch_ctk call now takes its CUDA version from ci/versions.yml or ci/test-matrix.yml, which is what makes failing closed practical. The sanitizer step installs compute-sanitizer from the build CTK (cuda.build.version) instead of the latest release found by ci/tools/guess_latest.sh, a run-time scrape of the redist server's directory listing, which this PR removes. That keeps the intent of CI: Install latest compute-sanitizer separately from CTK #594, a newer sanitizer for older toolkits, with a pinned version: in DO NOT MERGE: probe compute-sanitizer on Python 3.12 local-CTK rows #3400 the 12.9.1 and 13.0.2 rows segfaulted partway through the cuda.core tests with their own CTK's sanitizer and ran the full suite with the 13.5.0 one.

Checklist

I have read the Contributing Guidelines.
New or existing tests cover these changes.
The documentation is up to date with these changes.

Fetch cuda_sanitizer_api from the same pinned redistrib manifest as the rest of the mini CTK, rather than from whatever release ci/tools/guess_latest.sh scrapes from the redist directory listing. Remove guess_latest.sh, which has no other users.

copy-pr-bot · 2026-11-05T23:14:52Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

juenglin · 2026-11-05T23:18:23Z

@coderabbitai full review

coderabbitai · 2026-11-05T23:18:29Z

Action performed

Full review finished.

coderabbitai · 2026-11-05T23:23:20Z

No actionable comments were generated in the recent review.

Summary

Summary by CodeRabbit

Chores
- Automated Linux test runs use the CUDA version selected for each run, and sanitizer components can be added without replacing the existing CUDA installation.
- Removed the automatic lookup of the latest CUDA release; changes to that lookup path no longer select Linux-specific test infrastructure handling.
- Automated testing now requires verified checksums for public CUDA metadata and rejects versions without a pinned checksum.
- These changes affect testing workflows; end-user features and behavior remain unchanged.

The Linux wheel workflow now uses the build CTK version for sanitizer setup and adds the sanitizer component to the existing toolkit. Test-mode setup no longer computes LATEST_CUDA_VERSION; its helper script and Linux-specific CI path classification are removed. Public metadata verification now fails when a CUDA version has no pinned hash.

Changes: Sanitizer CUDA version and metadata

Layer / File(s)	Summary
Sanitizer setup and version selection `.github/actions/fetch_ctk/action.yml`, `.github/workflows/test-wheel-linux.yml`, `ci/tools/env-vars`, `ci/tools/guess_latest.sh`, `ci/tools/compute_ci_plan.py`, `ci/tools/tests/test_compute_ci_plan.py`	The action adds an `install-mode` input. In `add` mode, it preserves existing files and skips environment exports. The workflow uses the build CTK version and `add` mode for sanitizer setup. Test-mode setup no longer sets `LATEST_CUDA_VERSION`; the script that computed it and its Linux-specific path classification and test case are removed.
Pinned metadata verification `ci/tools/fetch_ctk_redistrib.py`, `ci/tools/tests/test_fetch_ctk_redistrib.py`, `ci/versions.yml`	CUDA 13.5.0 now has a pinned metadata SHA-256. Verification fails for public versions without a pinned hash. Tests check CI version coverage and digest format, and `ci/versions.yml` documents the pin requirement.

Change: Bug fix

Merge Risk: Minimal · up to d1470

The reviewed CI paths have no identified issue blocking merge; normal checks can proceed.

Architecture risk: Low · up to d1470

The change affects 1 system.

Changed systems: ci

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

observed — ci (service) was modified; 7 changed files map to changed impact.

Before / after behavior

observed — Modified behavior in ci/tools/env-vars: Removed the test-mode assignment that ran ci/tools/guess_latest.sh with TEST_CUDA_MAJOR and appended its result as LATEST_CUDA_VERSION to GITHUB_ENV.
observed — Modified behavior in ci/tools/guess_latest.sh: The ci/tools/guess_latest.sh script was removed, including its one-argument validation, directory fetch, CUDA redistribution filename matching and version sorting, and its output/error behavior for matching and missing files.
observed — Modified behavior in ci/tools/compute_ci_plan.py: Removed the ci/tools/guess_latest.sh → linux entry from TEST_INFRA_PLATFORMS.
observed — Modified behavior in ci/tools/fetch_ctk_redistrib.py: Updated the pinning instructions to state that verify-metadata fails for unpinned public versions and that hashes should be added alongside versions in the CI matrix.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Recent review info

Run configuration

Source mode: Auto

Repository: NVIDIA/cuda-python

Branch: main

File: .coderabbit.yaml

Commit: 8bcde89

Layers: Repository YAML

Review profile: CHILL

Plan: Enterprise

Run ID: `5075d55c-2728-4ce7-b2a0-f824f01b67fc`

Commits

Reviewing files that changed from the base of the PR and between `8fbed11` and `d14706c`.

Files selected for processing: 2

`.github/workflows/test-wheel-linux.yml`

`ci/versions.yml`

Files skipped from review as they are similar to previous changes: 1

ci/versions.yml

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

verify-metadata used to warn and pass when a public-server CUDA version had no pinned manifest hash, so the 13.4.2 -> 13.5.0 bump ran unpinned. Fail closed instead, with an explicit fetch_ctk allow-unpinned input (and --allow-unpinned flag) as the opt-out; it never overrides a mismatch against an existing pin. Manifests from an alternate redist server stay warn-only. Pin 13.5.0 and add a test that every CUDA version in ci/versions.yml and every LOCAL_CTK row of ci/test-matrix.yml is pinned.

No caller sets it, and pinning a new CUDA version is a one-line addition to KNOWN_REDISTRIB_SHA256 that the error message points to. Remove the fetch_ctk input and the --allow-unpinned flag; fetch_ctk/action.yml is back to its main-branch form.

Note in ci/versions.yml that each CUDA version there and in LOCAL_CTK matrix rows needs a pin in KNOWN_REDISTRIB_SHA256, and make the coverage test's failure message say where to add the pin and how to compute it.

juenglin · 2026-11-05T23:40:09Z

/ok to test bc92709

…t-sanitizer

juenglin · 2026-11-06T00:00:15Z

/ok to test 27f1b2f

github-actions · 2026-11-06T00:17:45Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

rwgk · 2026-11-06T02:21:51Z

Simplification: the sanitizer step installs compute-sanitizer from the CTK under test instead of the latest release

This reverts an earlier decision: PR 594 deliberately selected newer sanitizers to obtain bug fixes while testing older toolkits. A separately pinned sanitizer version would preserve that original intent without runtime scraping. Did you consider that already? It seems preferable to me.

NVIDIA#2437 made fetch_ctk empty cuda-path before every install. The sanitizer step runs fetch_ctk a second time for cuda_sanitizer_api alone, so it unintentionally deleted the mini CTK installed just before it, and the instrumented test runs failed before any test ran. NVIDIA#2437 then disabled the only matrix rows that run the sanitizer. Add an install-mode input. "add" installs on top of the existing mini CTK at the same cuda-path and keeps the environment the first call exported; the sanitizer step uses it.

With the matrix CTK's own sanitizer, the 12.9.1 and 13.0.2 rows segfaulted partway through the cuda.core tests. With the 13.5.0 sanitizer, both ran the full suite (NVIDIA#3400). This keeps the intent of NVIDIA#594 (newer sanitizer fixes for older toolkits) without the run-time scrape: the version comes from cuda.build.version in ci/versions.yml, which is already pinned.

juenglin · 2026-11-06T19:06:46Z

Simplification: the sanitizer step installs compute-sanitizer from the CTK under test instead of the latest release

... A separately pinned sanitizer version would preserve that original intent without runtime scraping. Did you consider that already? It seems preferable to me.

Good call, thanks. I tested both options in #3400 on Python 3.12 local-CTK rows. With each CTK's own sanitizer, 12.9.1 and 13.0.2 segfaulted partway through the cuda.core tests. With the 13.5.0 sanitizer, both ran the full suite. The sanitizer step now uses the build CTK version (cuda.build.version in ci/versions.yml). That keeps #594's intent with a pinned version, and it moves together with the build CTK.

juenglin · 2026-11-06T19:06:59Z

/ok to test d14706c

rwgk

LGTM

jpascucci-nv

no actionable comments from sol 6.1 high.

Adding cuda_gdb to fetch_ctk's default cuda-components made every Linux job download ~100 MB it does not use (build, docs, coverage, sdist, and test jobs that do not run the cuda-gdb test), and busted the fetch_ctk cache key for all of them. Revert the default-list change. Instead, mirror what NVIDIA#3397 did for cuda_sanitizer_api: a second fetch_ctk call with install-mode: add and cuda-components: "cuda_gdb", gated on LOCAL_CTK == '1' && TEST_CORE == 'true' so only local-CTK jobs that run the cuda.core test fetch it. cuda-gdb has no Windows build, so test-wheel-windows.yml is unchanged. Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>

Removed preview folders for the following PRs: - PR #2951 - PR #3314 - PR #3315 - PR #3371 - PR #3392 - PR #3396 - PR #3397 - PR #3400

juenglin added this to the CI milestone Nov 5, 2026

juenglin added CI/CD CI/CD infrastructure github_actions Pull requests that update GitHub Actions code labels Nov 5, 2026

juenglin self-assigned this Nov 5, 2026

juenglin added 3 commits November 5, 2026 15:31

CI: Point CTK version bumps at the redistrib manifest pins

bc92709

Note in ci/versions.yml that each CUDA version there and in LOCAL_CTK matrix rows needs a pin in KNOWN_REDISTRIB_SHA256, and make the coverage test's failure message say where to add the pin and how to compute it.

juenglin requested a review from rwgk November 5, 2026 23:40

juenglin assigned isVoid and unassigned isVoid Nov 5, 2026

juenglin requested a review from isVoid November 5, 2026 23:41

Merge remote-tracking branch 'upstream/main' into juenglin/drop-lates…

27f1b2f

…t-sanitizer

juenglin marked this pull request as ready for review November 6, 2026 00:12

juenglin enabled auto-merge (squash) November 6, 2026 00:26

juenglin mentioned this pull request Nov 6, 2026

DO NOT MERGE: probe compute-sanitizer on Python 3.12 local-CTK rows #3400

Closed

1 of 3 tasks

juenglin added the bug Something isn't working label Nov 6, 2026

juenglin changed the title ~~CI: Use the matrix CTK's compute-sanitizer instead of the latest release~~ CI: Fix compute-sanitizer setup and harden mini-CTK fetching Nov 6, 2026

juenglin requested a review from jpascucci-nv November 6, 2026 19:43

rwgk approved these changes Nov 6, 2026

View reviewed changes

juenglin disabled auto-merge November 6, 2026 19:46

juenglin enabled auto-merge (squash) November 6, 2026 19:46

jpascucci-nv approved these changes Nov 6, 2026

View reviewed changes

juenglin mentioned this pull request Nov 6, 2026

[BUG]: cuda.core tests fail under compute-sanitizer memcheck (use-after-free reports, kernelNsysProfile timeout) #3401

Open

1 task done

juenglin merged commit 63efb2d into NVIDIA:main Nov 6, 2026
120 checks passed

juenglin mentioned this pull request Nov 6, 2026

ci: install cuda-gdb for cuda.core local-CTK tests #3371

Merged

3 tasks done

github-actions Bot pushed a commit that referenced this pull request Nov 7, 2026

Clean up PR preview folders for 8 closed/merged PRs

e893c07

Removed preview folders for the following PRs: - PR #2951 - PR #3314 - PR #3315 - PR #3371 - PR #3392 - PR #3396 - PR #3397 - PR #3400

Conversation

juenglin commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

copy-pr-bot Bot commented Nov 5, 2026

Uh oh!

juenglin commented Nov 5, 2026

Uh oh!

coderabbitai Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

juenglin commented Nov 5, 2026

Uh oh!

juenglin commented Nov 6, 2026

Uh oh!

github-actions Bot commented Nov 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rwgk commented Nov 6, 2026

Uh oh!

juenglin commented Nov 6, 2026

Uh oh!

juenglin commented Nov 6, 2026

Uh oh!

rwgk left a comment

Choose a reason for hiding this comment

Uh oh!

jpascucci-nv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

juenglin commented Nov 5, 2026 •

edited

Loading

coderabbitai Bot commented Nov 5, 2026 •

edited

Loading

coderabbitai Bot commented Nov 5, 2026 •

edited

Loading

github-actions Bot commented Nov 6, 2026 •

edited

Loading