Skip to content

Kill hung tensor tests with pytest-timeout in the oneMath workflow - #3177

Merged
antonwolfy merged 2 commits into
masterfrom
onemath-tensor-test-timeout
Nov 11, 2026
Merged

antonwolfy merged 2 commits into
masterfrom
onemath-tensor-test-timeout

Conversation

@antonwolfy

Copy link
Copy Markdown
Contributor

#3152 added a per-test timeout to the tensor tests of the conda package workflows, for the hang at 99% (IntelPython/dpctl#1954). check-onemath.yaml hits the same hang but was left out, because it runs the tests without xdist.

Measured on check-onemath.yaml runs since 2026-10-01, from the retry annotations: 55 of 577 test jobs hit the 10 min timeout of the tensor step, 8 of them on two attempts, so about 10.5 h of runner time went to waiting. A healthy tensor step takes a median of 291 s (37 recent jobs).

This PR:

  • runs the tensor tests with --timeout=120 --timeout-method=thread, as conda-package.yml does;
  • adds -n auto to them: without xdist the thread method exits the whole pytest process at the first timeout, so the run stops there and the retry starts from scratch. With xdist only the stuck worker is killed and replaced, the run finishes, and the hung test is reported by name;
  • adds pytest-timeout and pytest-xdist to environments/test.yml. generate_coverage.yaml also merges that file but passes neither option, so its run doesn't change.

Checked locally with a test that hangs: a serial run with the thread method ends at the hang, the tests after it don't run and no cache is written, while with -n 2 the worker is replaced and the other tests all run.

Expected effect: a hang fails one test after 2 min instead of stalling the step for 10 min, and xdist should also shorten the tensor step; this PR's CI shows by how much.

  • Have you provided a meaningful PR description?
  • Have you added a test, reproducer or referred to an issue with a reproducer?
  • Have you tested your changes locally for CPU and GPU devices?
  • Have you made sure that new changes do not introduce compiler warnings?
  • Have you checked performance impact of proposed changes?
  • Have you added documentation for your changes, if necessary?
  • Have you added your changes to the changelog?

The oneMath workflow hits the same tensor-test hang at 99% (IntelPython/dpctl#1954) as the conda package workflows: since 2026-10-01, 55 of 577 test jobs hit the 10 min retry timeout of the tensor step, 8 of them twice, about 10.5 h of runner time spent waiting.

The tensor tests now run with `--timeout=120 --timeout-method=thread`, as in `conda-package.yml`, and with `-n auto`: without xdist the thread method exits the whole pytest process on the first timeout, while with xdist only the stuck worker is killed and replaced, the run finishes and the hung test is reported by name. `pytest-timeout` and `pytest-xdist` are added to `environments/test.yml`, which the coverage workflow also merges; it passes neither option, so its run doesn't change.
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Array API standard conformance tests for dpnp=0.22.0dev1=np2py314h8d9cdd5_14 ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

View rendered docs @ https://intelpython.github.io/dpnp/index.html

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the oneMath tensor-test timeout and parallel-execution changes.

No open issues 🎉

What changed in this PR

Extends the existing tensor-test hang mitigation to the oneMath workflow, using per-test timeouts and parallel workers so a stuck test can be terminated without stopping the entire test run.

Changes:

  • Applies a 120-second thread-based timeout to all four tensor-test commands, including retry paths.
  • Enables pytest-xdist with -n auto so timed-out workers can be replaced.
  • Adds the required pytest plugins to the shared test environment.
File Description
environments/​test.yml Adds pytest-timeout and pytest-xdist dependencies.
.github/​workflows/​check-onemath.yaml Configures and applies per-test timeouts and parallel execution for tensor tests.

Review effort: Balanced

@coveralls

Copy link
Copy Markdown
Collaborator

Coverage Status

coverage: 84.397%. remained the same — onemath-tensor-test-timeout into master

@antonwolfy antonwolfy added this to the 0.22.0 release milestone Nov 8, 2026
@antonwolfy antonwolfy self-assigned this Nov 8, 2026

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Thank you @antonwolfy

@antonwolfy
antonwolfy merged commit 98634e2 into master Nov 11, 2026
41 of 46 checks passed
@antonwolfy
antonwolfy deleted the onemath-tensor-test-timeout branch November 11, 2026 16:52
github-actions Bot added a commit that referenced this pull request Nov 11, 2026
…3177)

#3152 added a per-test timeout to the tensor tests of the conda package
workflows, for the hang at 99%
(IntelPython/dpctl#1954). `check-onemath.yaml`
hits the same hang but was left out, because it runs the tests without
xdist.

Measured on `check-onemath.yaml` runs since 2026-10-01, from the retry
annotations: 55 of 577 test jobs hit the 10 min timeout of the tensor
step, 8 of them on two attempts, so about 10.5 h of runner time went to
waiting. A healthy tensor step takes a median of 291 s (37 recent jobs).

This PR:
- runs the tensor tests with `--timeout=120 --timeout-method=thread`, as
`conda-package.yml` does;
- adds `-n auto` to them: without xdist the thread method exits the
whole pytest process at the first timeout, so the run stops there and
the retry starts from scratch. With xdist only the stuck worker is
killed and replaced, the run finishes, and the hung test is reported by
name;
- adds `pytest-timeout` and `pytest-xdist` to `environments/test.yml`.
`generate_coverage.yaml` also merges that file but passes neither
option, so its run doesn't change.

Checked locally with a test that hangs: a serial run with the thread
method ends at the hang, the tests after it don't run and no cache is
written, while with `-n 2` the worker is replaced and the other tests
all run.

Expected effect: a hang fails one test after 2 min instead of stalling
the step for 10 min, and xdist should also shorten the tensor step; this
PR's CI shows by how much.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants