Repository navigation
Kill hung tensor tests with pytest-timeout in the oneMath workflow - #3177
Merged
Merged
Conversation
The oneMath workflow hits the same tensor-test hang at 99% (IntelPython/dpctl#1954) as the conda package workflows: since 2026-10-01, 55 of 577 test jobs hit the 10 min retry timeout of the tensor step, 8 of them twice, about 10.5 h of runner time spent waiting. The tensor tests now run with `--timeout=120 --timeout-method=thread`, as in `conda-package.yml`, and with `-n auto`: without xdist the thread method exits the whole pytest process on the first timeout, while with xdist only the stuck worker is killed and replaced, the run finishes and the hung test is reported by name. `pytest-timeout` and `pytest-xdist` are added to `environments/test.yml`, which the coverage workflow also merges; it passes neither option, so its run doesn't change.
antonwolfy
requested review from
ndgrigorian and
vlad-perevezentsev
as code owners
November 8, 2026 18:22
2 of 7 tasks
Contributor
|
Array API standard conformance tests for dpnp=0.22.0dev1=np2py314h8d9cdd5_14 ran successfully. |
Contributor
|
View rendered docs @ https://intelpython.github.io/dpnp/index.html |
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the oneMath tensor-test timeout and parallel-execution changes.
No open issues 🎉
What changed in this PR
Extends the existing tensor-test hang mitigation to the oneMath workflow, using per-test timeouts and parallel workers so a stuck test can be terminated without stopping the entire test run.
Changes:
- Applies a 120-second thread-based timeout to all four tensor-test commands, including retry paths.
- Enables pytest-xdist with
-n autoso timed-out workers can be replaced. - Adds the required pytest plugins to the shared test environment.
| File | Description |
|---|---|
environments/test.yml |
Adds pytest-timeout and pytest-xdist dependencies. |
.github/workflows/check-onemath.yaml |
Configures and applies per-test timeouts and parallel execution for tensor tests. |
Review effort: Balanced
Collaborator
vlad-perevezentsev
approved these changes
Nov 11, 2026
vlad-perevezentsev
left a comment
Contributor
There was a problem hiding this comment.
LGTM
Thank you @antonwolfy
github-actions Bot
added a commit
that referenced
this pull request
Nov 11, 2026
…3177) #3152 added a per-test timeout to the tensor tests of the conda package workflows, for the hang at 99% (IntelPython/dpctl#1954). `check-onemath.yaml` hits the same hang but was left out, because it runs the tests without xdist. Measured on `check-onemath.yaml` runs since 2026-10-01, from the retry annotations: 55 of 577 test jobs hit the 10 min timeout of the tensor step, 8 of them on two attempts, so about 10.5 h of runner time went to waiting. A healthy tensor step takes a median of 291 s (37 recent jobs). This PR: - runs the tensor tests with `--timeout=120 --timeout-method=thread`, as `conda-package.yml` does; - adds `-n auto` to them: without xdist the thread method exits the whole pytest process at the first timeout, so the run stops there and the retry starts from scratch. With xdist only the stuck worker is killed and replaced, the run finishes, and the hung test is reported by name; - adds `pytest-timeout` and `pytest-xdist` to `environments/test.yml`. `generate_coverage.yaml` also merges that file but passes neither option, so its run doesn't change. Checked locally with a test that hangs: a serial run with the thread method ends at the hang, the tests after it don't run and no cache is written, while with `-n 2` the worker is replaced and the other tests all run. Expected effect: a hang fails one test after 2 min instead of stalling the step for 10 min, and xdist should also shorten the tensor step; this PR's CI shows by how much.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#3152 added a per-test timeout to the tensor tests of the conda package workflows, for the hang at 99% (IntelPython/dpctl#1954).
check-onemath.yamlhits the same hang but was left out, because it runs the tests without xdist.Measured on
check-onemath.yamlruns since 2026-10-01, from the retry annotations: 55 of 577 test jobs hit the 10 min timeout of the tensor step, 8 of them on two attempts, so about 10.5 h of runner time went to waiting. A healthy tensor step takes a median of 291 s (37 recent jobs).This PR:
--timeout=120 --timeout-method=thread, asconda-package.ymldoes;-n autoto them: without xdist the thread method exits the whole pytest process at the first timeout, so the run stops there and the retry starts from scratch. With xdist only the stuck worker is killed and replaced, the run finishes, and the hung test is reported by name;pytest-timeoutandpytest-xdisttoenvironments/test.yml.generate_coverage.yamlalso merges that file but passes neither option, so its run doesn't change.Checked locally with a test that hangs: a serial run with the thread method ends at the hang, the tests after it don't run and no cache is written, while with
-n 2the worker is replaced and the other tests all run.Expected effect: a hang fails one test after 2 min instead of stalling the step for 10 min, and xdist should also shorten the tensor step; this PR's CI shows by how much.