Build and test conda packages and wheels by jharlow-intel · Pull Request #3151 · IntelPython/dpnp

jharlow-intel · 2026-11-03T17:56:24Z

Adds ci/ and .github/workflows/intel-ci.yml, which build the conda packages and wheels for Linux and Windows on every declared Python, and test each in a clean environment.

conda-recipe/ is now the only recipe: the conda-forge one that was conda-recipe-cf/. It builds against conda-forge's dpctl 0.23.0, which has no 3.10, so its Pythons start at 3.11. The Intel-channel recipe, with rattler_recipe.yaml, and conda-package.yml are removed; that workflow's Array API conformance and pybind11 examples jobs move to conda-package-cf.yml.
The dppy/label/dev uploads and their cleanup move into intel-ci.yml, for pushes to master and tags only.
Wheels build with cibuildwheel and icx 2026.2 against the dpctl wheel on PyPI, and require the oneAPI runtime wheels (dpcpp-cpp-rt, intel-sycl-rt, mkl, onemkl-sycl-*), as the PyPI wheels do. PyPI has no dpctl for 3.15 yet, so the 3.15 classifier is dropped until it does.
ci/test.py runs dpnp.tests, then dpnp.tests.tensor, against the installed package, and fails when it finds no SYCL device: the OpenCL CPU device in Actions, or a GPU with GH_CI_DEVICE=gpu.

The required checks on master come from conda-package.yml, so they need replacing with this workflow's, such as Test Linux Python 3.11, before this can merge.

Validation, with a temporary push trigger on this branch: run 41672888041 built and tested the conda packages for 3.11 and 3.15 and the wheels for 3.10 and 3.14, on Linux and Windows, and conda-package-cf.yml, pre-commit and zizmor passed. An earlier full-matrix run passed every declared Python. Not run yet: a GPU, and the uploads, which only master and tags do.

github-actions · 2026-11-09T15:29:18Z

View rendered docs @ https://intelpython.github.io/dpnp/pull/3151/index.html

github-actions · 2026-11-09T16:07:53Z

Array API standard conformance tests for dpnp=0.22.0dev1+11.g24c267f0d ran successfully.
Passed: 1491
Failed: 0
Skipped: 6

coveralls · 2026-11-09T16:26:09Z

coverage: 84.424%. remained the same — ci/intel-python-ci into master

vlad-perevezentsev · 2026-11-09T16:26:10Z

+      - name: Build and run the recipe tests
+        env:
+          PYTHON_VERSION: ${{ matrix.python }}
+        run: python ci/build.py --python "$PYTHON_VERSION" --output "$RUNNER_TEMP/output"


The conda packages and the wheels now build and test only against released dpctl (conda-forge, PyPI); the removed workflow used dppy/label/dev first. So dpnp can no longer require a dpctl dev build: dpctl>=0.24.0dev0, as #3169 sets in pyproject.toml, makes the wheel install fail, and the same in the recipe fails the conda build, as only 0.23.0 is released. Is that intended? If not, build.py --channel dppy/label/dev --channel conda-forge and the same channel in hooks.py would keep it.

vlad-perevezentsev · 2026-11-09T16:26:10Z

+
+    Returns its exit status, or None if it hung on every attempt.
+    """
+    for attempt in range(1, ATTEMPTS + 1):


This retries a suite only when it hangs. The removed workflow reran any failed suite (2 attempts, 3 for the tensor tests) and stopped a stuck tensor test after 120 s with pytest-timeout (IntelPython/dpctl#1954). Now one flaky failure fails the job and blocks the upload, and one stuck tensor test costs 20 minutes. Could it rerun on any failure, and keep --timeout=120 --timeout-method=thread for dpnp.tests.tensor?

Done: a failed or hung suite now runs up to 2 times for the main tests and 3 times for the tensor tests, as before, and dpnp.tests.tensor runs with --timeout=120 --timeout-method=thread. pytest-timeout is in the test requirements.

vlad-perevezentsev · 2026-11-09T16:26:10Z

-      - '*DPCTLSyclInterface*'
-      - '*/dpnp_backend_c.dll'        # [win]
+  number: {{ GIT_DESCRIBE_NUMBER }}
+  ignore_run_exports:


The removed rattler_recipe.yaml ignored the mkl-dpcpp run export of mkl-devel-dpcpp, and this recipe does not. conda-forge's mkl-devel-dpcpp 2026.2.0 exports mkl-dpcpp >=2026.2.0,<2027.0a0, which pulls in onemkl-sycl-datafitting and onemkl-sycl-stats. dpnp uses neither, and these packages now go to dppy/label/dev. Could - mkl-dpcpp go back into ignore_run_exports?

Right, mkl-devel-dpcpp 2026.2.0 run-exports mkl-dpcpp >=2026.2.0,<2027.0a0. mkl-dpcpp is back in ignore_run_exports; the run requirements still list the onemkl-sycl-* packages dpnp uses.

vlad-perevezentsev · 2026-11-09T16:26:10Z

+
+  examples:
+    name: Test examples
+    needs: linux


needs: linux waits for the whole reusable workflow, tests included: a failing Linux test now skips the examples and the Array API comment, which needed only the build before. The examples also lost 3.14t. As this workflow now repeats intel-ci.yml's builds for 3.12-3.14, could both jobs move there, with needs: build and a package every event builds (3.14t or 3.15)?

Moved both into intel-ci.yml with needs: [matrix, build], so they wait only for the builds. The examples run on the newest regular and free-threaded Linux packages, 3.15 and 3.14t today, which every event builds, and the Array API job runs on the newest regular one. matrix.py --newest --platform linux picks them, so no version is hardcoded in the workflow. Both install with --strict-channel-priority, so dpnp can only come from the package that was built. With that, conda-package-cf.yml and conda-build-test.yml only repeated intel-ci.yml's builds, so they're removed, and the README badge points at intel-ci.yml.

vlad-perevezentsev · 2026-11-09T16:26:10Z

+KEEP = (
+    "PATH",
+    "HOME",
+    "USER",
+    "LANG",
+    "TERM",
+    "TMPDIR",
+    "TZ",
+    "http_proxy",
+    "https_proxy",
+    "no_proxy",
+    "HTTP_PROXY",
+    "HTTPS_PROXY",
+    "NO_PROXY",
+    "SSL_CERT_FILE",
+    "REQUESTS_CA_BUNDLE",
+    "CONDA_PKGS_DIRS",
+    "SYSTEMROOT",
+    "SYSTEMDRIVE",
+    "WINDIR",
+    "COMSPEC",
+    "PATHEXT",
+    "TEMP",
+    "TMP",
+    "USERPROFILE",
+    "APPDATA",
+    "LOCALAPPDATA",
+    "PROGRAMDATA",
+    "NUMBER_OF_PROCESSORS",
+    "PROCESSOR_ARCHITECTURE",
+)


test_env keeps only KEEP and the GH_CI_* values it sets, so GH_CI_DEVICE never reaches test.py. On a GPU runner (vars.GH_CI_RUNNER) with a broken GPU driver the tests then pass on the CPU device, which ci/README.md says cannot happen. ONEAPI_DEVICE_SELECTOR is dropped too. Could both go into KEEP?

Fixed: GH_CI_DEVICE and ONEAPI_DEVICE_SELECTOR are in KEEP now, so a GPU run without a working GPU fails, as ci/README.md says.

vlad-perevezentsev · 2026-11-09T16:26:10Z

 .. code-block:: bash

-    conda-build conda-recipe -c https://software.repos.intel.com/python/conda/ -c conda-forge --override-channels
+    conda-build conda-recipe -c conda-forge --override-channels


conda_build_config.yaml now lists six Python versions, so this command builds six packages, about six times the old build time. Adding --python 3.14 here would keep it to one.

Added, with the ABI spelled out, --python "3.14.* *_cp314", as a bare 3.14 can resolve to the free-threaded build.

vlad-perevezentsev · 2026-11-09T16:26:10Z

 | --- | --- | --- |
 | pre-commit (`Check`) | `.github/workflows/pre-commit.yml` | yes |
-| conda build and tests, Linux and Windows, py3.13-3.15 and 3.14t (`Test (...)`) | `conda-package.yml` | yes |
+| conda packages and wheels built and tested, Linux and Windows, py3.11-3.15 and 3.14t for conda, py3.10-3.14 and 3.14t for wheels (`Test <Platform> Python <version>`, `Test <Platform> Wheels Python <version>`); the conda matrix is `conda-recipe/conda_build_config.yaml`, the wheel matrix the Python classifiers in `pyproject.toml`; see `ci/README.md` | `intel-ci.yml` | no |


The wheels are 3.11-3.14 and 3.14t: pyproject.toml has no 3.10 classifier, as #3173 dropped 3.10. Also, the old row was required on master. Is a required check planned for intel-ci.yml, or should branch protection just drop the old Test (...) checks?

Fixed the wheel range to 3.11-3.14 and 3.14t. The required Test (...) checks come from conda-package.yml, which this PR removes, so branch protection needs them swapped for intel-ci.yml's jobs before this can merge.

vlad-perevezentsev · 2026-11-09T16:26:10Z

@@ -0,0 +1,655 @@
+"""


matrix.py, build.py, hooks.py and benchmark.py have no license header, while test.py and install_icx.py have one. AGENTS.md asks for the BSD-3 header from a neighbouring file on every new file.

Added the BSD-3 header from test.py to matrix.py, build.py, hooks.py and benchmark.py.

vlad-perevezentsev · 2026-11-09T16:26:10Z

+workflow: intel-ci.yml
+artifacts:
+  conda: "dpnp {platform} Python {python}"
+  wheel: "dpnp {platform} Wheels Python {python}"
+test:
+  requires: [pytest, pytest-xdist, scipy]
+  # The wheels' requirements have no OpenCL CPU runtime of their own, which the tests need
+  wheel_requires: [pytest, pytest-xdist, scipy, intel-opencl-rt]
+benchmark:
+  # PyYAML, which asv needs, has no PyPI wheel for every Python the recipe declares.
+  requires: [scipy, pyyaml]
+  # conda-forge lacks asv for some Python the recipe declares; the wheel on PyPI fits all.
+  pip_requires: [asv]


Nit: Nothing here runs benchmark.py or reads workflow, artifacts and benchmark from this file, and the JUnit reports test.py writes are not uploaded. Are they for another Intel CI service, and is ci/ copied from a shared template (it also has rattler-build, macOS and conda-recipe-cf paths dpnp does not use)? A line in ci/README.md would help either way.

Added a paragraph to ci/README.md: intel-ci.yml doesn't read those keys, run benchmark.py or keep the JUnit reports; they're there to run the same build, tests and benchmarks outside this workflow. The rattler-build, macOS and conda-recipe-cf/ handling is general support in the scripts that dpnp doesn't use.

vlad-perevezentsev · 2026-11-09T16:26:10Z

 * Moved `blas`, `fft`, `lapack`, `sparse` and `vm` extensions into `dpnp/libtensor/extensions` [#3131](https://github.com/IntelPython/dpnp/pull/3131)
 * Moved the libtensor headers shipped with dpnp from `dpnp/tensor/libtensor/include` to `dpnp/libtensor/include`, and changed `dpnp.get_include` to return that directory [#3131](https://github.com/IntelPython/dpnp/pull/3131)
 * Improved performance of `dpnp.scipy.sparse.csr_matrix.sort_indices` by sorting once on a combined row and column key instead of in two passes [#3134](https://github.com/IntelPython/dpnp/pull/3134)
+* Changed the CI to build and test the conda packages of `conda-recipe` and the wheels with Intel CI (`ci/`, `.github/workflows/intel-ci.yml`); the conda packages now require `onemkl-sycl-sparse` and the wheels declare the oneAPI runtime wheels in their metadata [#3151](https://github.com/IntelPython/dpnp/pull/3151)


Nit: The dev packages from the removed recipe already required onemkl-sycl-sparse. For their users the change is that the dppy/label/dev packages are now built against conda-forge only; could the line say that instead?

Reworded as suggested.

Adds a ci/ directory and .github/workflows/intel-ci.yml, which build the conda packages and the wheels for every declared platform and Python and test each with ci/test.py. A pull request builds the oldest and the newest regular Python and the newest free-threaded one, and every push and tag run gets a concurrency group of its own. conda-recipe/ is now the only recipe, the conda-forge one that was conda-recipe-cf/. The Intel-channel recipe and conda-package.yml are removed, and so are conda-package-cf.yml and its conda-build-test.yml, whose builds intel-ci.yml repeats. The dev uploads to anaconda.org, the Array API conformance job and the pybind11 examples move to intel-ci.yml. The examples and the Array API job wait only for the builds of the newest regular and free-threaded Linux packages, which every event builds, and install dpnp from the built packages only. The benchmarks take PyYAML from conda-forge, as PyPI has no Python 3.15 wheel for it yet. ci/build.py refuses two built packages with the same file name, which would overwrite each other in its flat output directory, and takes METADATA's line ending from its headers. conda-build records the commit each conda package came from in info/about.json, and LDFLAGS links the Linux wheels' libraries with full RELRO. ci/test.py reruns a failed suite, twice for the main tests and three times for the tensor tests, and stops a tensor test stuck for 120 s. GH_CI_DEVICE and ONEAPI_DEVICE_SELECTOR reach the tests. The recipe ignores mkl-dpcpp's run export, and the ci/ scripts carry the BSD-3 header.

jharlow-intel self-assigned this Nov 3, 2026

jharlow-intel requested review from antonwolfy, ndgrigorian and vlad-perevezentsev as code owners November 3, 2026 17:56

jharlow-intel force-pushed the ci/intel-python-ci branch from 73735c3 to f70dbd8 Compare November 3, 2026 17:57

jharlow-intel changed the title ~~Build and test conda packages and wheels for Intel Python CI~~ Build and test conda packages and wheels Nov 5, 2026

jharlow-intel force-pushed the ci/intel-python-ci branch 2 times, most recently from 6aca46e to e9f220a Compare November 9, 2026 14:42

vlad-perevezentsev reviewed Nov 9, 2026

View reviewed changes

jharlow-intel force-pushed the ci/intel-python-ci branch from e9f220a to b53329b Compare November 9, 2026 21:51

jharlow-intel force-pushed the ci/intel-python-ci branch from b53329b to c93fd5e Compare November 10, 2026 22:49

Conversation

jharlow-intel commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Nov 9, 2026

Uh oh!

github-actions Bot commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Nov 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jharlow-intel commented Nov 3, 2026 •

edited

Loading

github-actions Bot commented Nov 9, 2026 •

edited

Loading

coveralls commented Nov 9, 2026 •

edited

Loading