You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ci: install cuda-gdb for cuda.core local-CTK tests - #3371
Follow-up to #3397. Enables cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_lines, which currently always skips in mini-CTK jobs because cuda-gdb is not on PATH.
Rather than adding cuda_gdb to fetch_ctk's default cuda-components list (the original approach on this branch), which would make every Linux job download ~100 MB it does not use and bust the fetch_ctk cache key for all of them, this PR mirrors what #3397 did for cuda_sanitizer_api: a second fetch_ctk call in test-wheel-linux.yml with install-mode: add and cuda-components: "cuda_gdb", gated on LOCAL_CTK == '1' && TEST_CORE == 'true' so only local-CTK jobs that run the cuda.core test fetch it. The default mini CTK installed by build, docs, coverage, and sdist jobs is unchanged.
cuda_gdb has no Windows build, and fetch_ctk already skips components a platform does not publish, so test-wheel-windows.yml is unchanged. Only matrix entries that use the mini CTK (LOCAL_CTK == '1') are affected; jobs that take CUDA from wheels do not get cuda-gdb.
Tests enabled
cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_lines runs instead of skipping in local-CTK cuda.core jobs (standard and nightly-cuda-core). It debugs a kernel compiled with debug=True and checks that cuda-gdb shows the NVRTC source lines. It reports xfail rather than failing on runners where ptrace is not permitted.
cuda_pathfinder/tests/test_find_nvidia_binaries.py::test_find_binary_utilities[cuda-gdb] and [cuda-gdbserver] locate real binaries in jobs where cuda-gdb is installed (cuda.core local-CTK jobs that also run pathfinder). They pass either way, so this only adds coverage of the found path; in pathfinder-only jobs cuda-gdb is not installed and they continue to return None.
cuda_core's test_cuda_gdb_shows_nvrtc_debug_source_lines skips when
cuda-gdb is not on PATH, which is always the case with the mini CTK.
Adding cuda_gdb (about 100 MB, Linux only) to the default components lets
the test run in every Linux job that uses fetch_ctk's default list.
fetch_ctk already skips components a platform does not publish, so
Windows jobs are unchanged.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
No actionable comments were generated in the recent review.
Summary
Summary by CodeRabbit
Chores
Linux wheel testing now installs the CUDA debugger when local CUDA toolkit testing is enabled for core tests. Other test configurations are unchanged. This update affects the testing workflow and does not change the functionality available in the application or its public interfaces.
The Linux wheel-test workflow adds a conditional step that requests cuda_gdb through fetch_ctk when LOCAL_CTK is 1 and TEST_CORE is true. The step uses install-mode: add.
Changes: Linux wheel-test setup
Layer / File(s)
Summary
Add cuda_gdb to local CTK .github/workflows/test-wheel-linux.yml
When LOCAL_CTK is 1 and TEST_CORE is true, the workflow calls fetch_ctk to request cuda_gdb for the matrix CUDA version with install-mode: add.
Possibly related PRs
NVIDIA/cuda-python#3306: Changes fetch_ctk component download behavior used to fetch cuda_gdb.
Suggested labels:github_actions
Change: Other
Merge Risk: Moderate · up to f439d
Nightly tests using an older wheel build can fail during debugger setup. Use a compatible helper before merging.
Architecture risk: Low · up to f439d
The changed surface does not map to a changed system, dependency edge, entrypoint, or external dependency.
Changed systems: None identified.
Architecture concerns
No architecture-level concerns identified.
Review details
Before / after behavior
observed — Modified behavior in .github/workflows/test-wheel-linux.yml: Adds a conditional cuda_gdb setup step for local-CTK core tests. It calls the workflow-revision fetch_ctk action with the current host platform and matrix CUDA version, requests cuda_gdb, and sets install-mode: add.
- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Reviewing files that changed from the base of the PR and between 30cb707 and f439dbe.
Files selected for processing: 1
.github/workflows/test-wheel-linux.yml
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.
suggestion: Keep cuda_gdb out of the global default and add it only to the Linux mini-CTK invocation that needs the debugger. Callers that omit cuda-components inherit this list, so unrelated jobs download and cache the debugger archive on cold misses. This increases CI transfer and cache use for jobs that do not run the debugger test. (raw.githubusercontent.com)
Reviewing files that changed from the base of the PR and between deb3d04 and 724bc86.
Files selected for processing: 1
.github/actions/fetch_ctk/action.yml
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
The reason will be displayed to describe this comment to others. Learn more.
In numba-cuda-mlir, we overlay fetch_ctk call to additionally fetch cuda-gdb in the one pipeline that needs it. Let's do the same here (if it is needed at all).
Platform filtering and cache invalidation handle the added component correctly.
All 13 toolkit-helper tests pass locally; full CI is green.
The debugger source-line test passes in both x86_64 and aarch64 CI.
The remaining tradeoff is extra download and cache usage for Linux jobs that do not need the debugger, already acknowledged in the PR description.
CUDA-GDB history and download sizes
CUDA-GDB is a long-established tool. NVIDIA’s CUDA 2.1 release in January 2009 already included GDB-based CUDA debugging. It extends GDB to debug CPU and GPU code together, including breakpoints, stepping, variables, and GPU state. cuda_gdb is the redistribution component’s name; cuda-gdb is the executable. Release history, tool overview.
For the CUDA versions relevant to this PR, the compressed debugger downloads are:
These are decimal MB for this component alone. There is no native Windows package in those manifests, so this PR adds no Windows download. CUDA-GDB does support WSL2 using Linux tooling. Supported platforms.
Compiles a CUDA source string through cuda.core.Program and NVRTC with debug=True.
Loads and launches the resulting kernel, with the debugger configured to stop at kernel launch.
Asks the debugger to display source and checks for the generated .cu filename and a known source line.
The practical benefit is verifying that a user can see their runtime-compiled CUDA source in a real debugger. That requires the generated source file, compiler debug information, source paths, and GPU debugger to work together. Existing file-management tests cover parts of that chain; this exercises the complete interaction. Compilation and launch helper.
There is also modest additional coverage of finding installed cuda-gdb and cuda-gdbserver binaries. Those lookup tests still accept None, and nothing here exercises remote debugging through cuda-gdbserver. Lookup test.
In numba-cuda-mlir, we overlay fetch_ctk call to additionally fetch cuda-gdb in the one pipeline that needs it. Let's do the same here (if it is needed at all).
Q: Who/where is using this? Our "mini" CTK is not mini anymore because cuda-gdb pulls in its own Python...
The test cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_lines is currently never executed. Are you suggesting that we include cuda_gdb in the fetch for test workflows only? Or live with the gap?
Q: Who/where is using this? Our "mini" CTK is not mini anymore because cuda-gdb pulls in its own Python...
The test cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_lines is currently never executed. Are you suggesting that we include cuda_gdb in the fetch for test workflows only? Or live with the gap?
I checked CI log of this job, The test is executed there and passes:
What is the "mini CTK" used for? My agent says it tests the scenario of downloading and installing the CUDA Toolkit. If so, wouldn't "local CTK" be a more fitting name? "mini CTK" sounds like a subset, as if it runs only some of the tests rather than all of them.
What is the "mini CTK" used for? My agent says it tests the scenario of downloading and installing the CUDA Toolkit. If so, wouldn't "local CTK" be a more fitting name? "mini CTK" sounds like a subset, as if it runs only some of the tests rather than all of them.
Both names are already in use in the workflow code, and they're basically equivalent AFAIK.
"mini CTK" sounds like a subset
Correct. The way I think about it, roughly: it's a subset of what external users usually install into /usr/local/cuda
(In the CI, I think the directory is actually elsewhere.)
Adding cuda_gdb to fetch_ctk's default cuda-components made every Linux
job download ~100 MB it does not use (build, docs, coverage, sdist, and
test jobs that do not run the cuda-gdb test), and busted the fetch_ctk
cache key for all of them.
Revert the default-list change. Instead, mirror what NVIDIA#3397 did for
cuda_sanitizer_api: a second fetch_ctk call with install-mode: add and
cuda-components: "cuda_gdb", gated on LOCAL_CTK == '1' && TEST_CORE ==
'true' so only local-CTK jobs that run the cuda.core test fetch it.
cuda-gdb has no Windows build, so test-wheel-windows.yml is unchanged.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
juenglin
changed the title
ci: install cuda-gdb in the default fetch_ctk toolkit
ci: install cuda-gdb for cuda.core local-CTK tests
Nov 6, 2026
Reviewing files that changed from the base of the PR and between 724bc86 and 30cb707.
Files selected for processing: 1
.github/workflows/test-wheel-linux.yml
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.
Nightly runs check out an older inputs.sha, so ./ can resolve to a
fetch_ctk that predates install-mode. That copy would ignore
install-mode: add and empty the mini CTK. Use $/ so the action comes
from the workflow's commit.
Signed-off-by: Ralf Juengling <rjuengling@nvidia.com>
The reason will be displayed to describe this comment to others. Learn more.
I consulted with code GPT-6.1-Sol ultra:
Yes—your understanding is correct, and I prefer this revision. No new issues found at f439dbe.
The original mini-CTK contents and cache key stay unchanged.
Only Linux mini-CTK jobs testing cuda.core install the debugger, using a separate cache shared across Python/GPU configurations.
install-mode: add preserves the existing toolkit and environment. Updated setup.
The extra transfer remains per affected job, including cache hits. One current CI job restored the original toolkit cache, then downloaded a separate 117 MB debugger cache. That cached size differs slightly from NVIDIA’s archive because the action repackages it.
I consider that a reasonable cost for exercising real source debugging. The updated debugger test passes on both x86_64 and ARM64; full CI is still running. I’d keep this approach without further optimization unless measured setup time becomes a concern.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CI/CDCI/CD infrastructuretestImprovements or additions to tests
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
leofang
left a comment
rwgk
left a comment
Description
Follow-up to #3397. Enables
cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_lines, which currently always skips in mini-CTK jobs becausecuda-gdbis not on PATH.Rather than adding
cuda_gdbtofetch_ctk's defaultcuda-componentslist (the original approach on this branch), which would make every Linux job download ~100 MB it does not use and bust thefetch_ctkcache key for all of them, this PR mirrors what #3397 did forcuda_sanitizer_api: a secondfetch_ctkcall intest-wheel-linux.ymlwithinstall-mode: addandcuda-components: "cuda_gdb", gated onLOCAL_CTK == '1' && TEST_CORE == 'true'so only local-CTK jobs that run the cuda.core test fetch it. The default mini CTK installed by build, docs, coverage, and sdist jobs is unchanged.cuda_gdbhas no Windows build, andfetch_ctkalready skips components a platform does not publish, sotest-wheel-windows.ymlis unchanged. Only matrix entries that use the mini CTK (LOCAL_CTK == '1') are affected; jobs that take CUDA from wheels do not getcuda-gdb.Tests enabled
cuda_core/tests/test_program.py::test_cuda_gdb_shows_nvrtc_debug_source_linesruns instead of skipping in local-CTK cuda.core jobs (standard andnightly-cuda-core). It debugs a kernel compiled withdebug=Trueand checks thatcuda-gdbshows the NVRTC source lines. It reports xfail rather than failing on runners where ptrace is not permitted.cuda_pathfinder/tests/test_find_nvidia_binaries.py::test_find_binary_utilities[cuda-gdb]and[cuda-gdbserver]locate real binaries in jobs where cuda-gdb is installed (cuda.core local-CTK jobs that also run pathfinder). They pass either way, so this only adds coverage of the found path; in pathfinder-only jobs cuda-gdb is not installed and they continue to returnNone.Checklist