Build and test conda packages and wheels by jharlow-intel · Pull Request #204 · IntelPython/mkl_random

jharlow-intel · 2026-10-23T20:23:40Z

Adds ci/ and .github/workflows/intel-ci.yml, which build the conda packages for Linux, Windows and Intel macOS and the wheels for Linux and Windows, on every declared Python, and test each in a clean environment.

conda-recipe/ is now the only recipe: the conda-forge one that was conda-recipe-cf/. conda-package.yml and conda-package-cf.yml are removed, as intel-ci.yml builds and tests the recipe.
The conda packages build Python 3.10 to 3.15 plus 3.14t, from the python: list this adds to conda_build_config.yaml. The wheels follow the pyproject.toml classifiers: 3.10 to 3.15 plus 3.14t and 3.15t.
Wheels build with cibuildwheel, with MSVC on Windows, leave MKL out and require mkl from the mkl-devel version they were built with up to its next major, through [tool.gh-ci] pin-compatible, which ci/build.py adds to each wheel. The Linux wheels build with conda-forge's hardening flags, and ci/build.py refuses a wheel without full RELRO or with a library search path outside $ORIGIN.
meson.build adds Library to CMake's search path on Windows, where mkl-devel puts its CMake config.
ci/test.py runs pytest --pyargs mkl_random on the installed package. The tests' hashlib.md5 calls pass usedforsecurity=False.
Each test job runs for every package that built, even when another build failed. A pull request builds the oldest and newest Python plus the newest free-threaded one; master and tags build them all.
conda-build records the source commit in each package's info/about.json.
ci/benchmark.py runs the asv suite in benchmarks/ against the installed package: one quick sample of each benchmark on a pull request, a full measurement on a tag.

Miniforge is pinned to 26.7.2-0, as 26.9.4-0 makes setup-miniconda's conda init fail on Windows runners.

Validation: run 42669872840 on 75f0985 builds and tests the reduced pull request matrix on Linux, Windows and macOS. Not run yet: the full matrix, which only master and tags build.

antonwolfy · 2026-10-26T21:29:32Z

+
+concurrency:
+  group: intel-ci-${{ github.ref }}
+  cancel-in-progress: true


With cancel-in-progress: true keyed on github.ref, two quick merges cancel the first commit's whole run. Jenkins, which waits for that run, gets no packages for it.

Suggested change

cancel-in-progress: true

cancel-in-progress: ${{ github.event_name == 'pull_request' }}

Applied in 6006c16, so a push to master no longer stops the run before it. Every tag is its own ref, so tag runs never cancelled each other either.

antonwolfy · 2026-10-26T21:37:08Z

+numpy:
+    - '2'


That only will work when ##202 is merged

Agreed, this should land after #202. With the current pin_compatible('numpy', min_pin="x.x"), the 3.15 package built here requires numpy >=2.5,<3.0a0 at run time, which #202 fixes. Once #202 merges I will rebase: numpy: '2' will come from #202, and 3.10 will leave the python: list along with its matrix.

antonwolfy · 2026-10-26T21:45:27Z

@@ -1,3 +1,16 @@
+# The Python versions to build, one per line and without selectors: ci/matrix.py
+# reads them for .github/workflows/intel-ci.yml. NumPy 2, as in conda-forge's
+# global pinning; conda-package-cf.yml passes its own with --python and --numpy.


--numpy is dropping in #202

Removed from the comment in 6006c16, so it stays true after #202.

antonwolfy · 2026-10-26T21:47:12Z

+        "--artifacts", required=True, type=Path, help="the downloaded artifact"
+    )
+    parser.add_argument(
+        "--work", required=True, type=Path, help="an empty scratch directory"


hooks.py doesn't check that --work is empty, and may install conda-index into the user's base environment.

Fixed in b758764. hooks.py now refuses a --work that is not an empty directory, and it writes the local channel's repodata.json itself, so it installs nothing into base.

antonwolfy · 2026-10-26T21:48:47Z

+
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"


Is there any issue to use the newer python?

No issue: the job only runs ci/matrix.py, which needs nothing beyond the standard library. Moved to 3.14 in 6006c16.

antonwolfy · 2026-10-26T21:53:59Z

+          activate-environment: ""
+          conda-remove-defaults: "true"
+
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0


Alternatively we can drop setup-python and run with shell: bash -el {0} with base activated.

bash is available on Windows runners too, and setup-miniconda supports this shell there. The step bodies become plain python ci/build.py … and python ci/hooks.py …. The $CONDA path code and hooks.py's re-run of itself would then only matter for local use.

Done in 6006c16: build and test run in bash -el {0} with base active, without setup-python. On Windows the new steps work: conda-build installs, the package builds and the imports pass. This run is still red on Windows because conda-forge's pytest 9.2.0, published today, dropped colorama again, so the recipe's pip check fails (conda-forge/pytest-feedstock#208). I will re-run once a fixed build is out.

Re-ran after conda-forge published the fixed pytest build: the run is green on Linux and Windows, test jobs included.

antonwolfy · 2026-10-26T22:00:21Z

@@ -0,0 +1,135 @@
+# Intel Python CI (gh-ci) for mkl_random: builds the conda-forge recipe, conda-recipe-cf, for every


The same PR is open in three repos at once.
Have you considered moving common things (scripts, workflow) to some common repo like https://github.com/IntelPython/devops-tools ?
That would help to reduce the duplication.

Yes, we considered it. Duplicated code is a known consequence of the new CI's design, where each repository carries everything its CI needs, and the design's other benefits make up for it.

Build and test steps run in a bash login shell with conda base active, so they call ci/build.py and ci/hooks.py directly and need no setup-python. Pushes to the default branch no longer cancel each other. Same as IntelPython/mkl_random#204.

vchamarthi

Went through this on the branch. The PR description looks stale: conda-recipe-cf is deleted, macOS and the wheels are built too, and Jenkins is enabled in dry run. There are also no AGENTS.md notes in the diff. Also, conda-package.yml is deleted, so the Intel channel build is no longer tested anywhere. Details inline.

vchamarthi · 2026-11-04T19:17:50Z

      - wheel  >=0.41.3
    run:
      - python
+      - {{ pin_compatible('numpy', min_pin="x.x", max_pin="x") }}


pin_compatible('numpy', min_pin="x.x", max_pin="x") resolves to >=2.2,<3.0a0 / >=2.5,<3.0a0 in the build logs, depending on the build. That excludes numpy 2.0 to 2.4 at runtime for the first one, and the minimum follows whatever numpy was in the host env. Is that the intent? A fixed lower bound like the old >=1.26.4 would be easier to reason about.

Not intended, agreed. #202 fixed it on master, and this is rebased on it: the run requirement is a fixed numpy >=1.26.4, as in pyproject.toml, and numpy's run_exports add the upper bound.

vchamarthi · 2026-11-04T19:17:50Z

+
+  test:
+    name: Test ${{ matrix.name }} Python ${{ matrix.python }}
+    needs: [matrix, build]


needs: [matrix, build] means a macOS conda failure skips test and so blocks the release, same for the wheel job at :155. If macOS is meant to gate releases that is fine, just checking it is on purpose.

Yes, a package that fails to build on any platform, macOS included, fails the run. Skipping every test leg wasn't intended, though: in 75f0985 the test jobs use if: ${{ !cancelled() && needs.matrix.result == 'success' }}, as mkl-service does, so every package that built still gets tested and only the failed leg's test fails. Same for test-wheel.

vchamarthi · 2026-11-04T19:17:50Z

+  # macOS runs on Intel runners, as MKL has no Apple silicon build; Jenkins ignores it.
+  build:
+    name: ${{ matrix.name }} Python ${{ matrix.python }}
+    needs: matrix


This builds conda-recipe on every push and PR, and I don't see conda-package-cf.yml removed, so the same package may be built twice. Worth deleting or limiting the old workflow if it is now redundant.

Agreed, it was redundant. 75f0985 removes conda-package-cf.yml; intel-ci.yml builds and tests the recipe.

vchamarthi · 2026-11-04T19:17:50Z

+"""
+Intel Python CI (gh-ci) test for mkl_random.
+
+GitHub Actions, through ci/hooks.py, and Intel's Jenkins both run this with one


Nit: the docstring says the test runs against a conda package, but the wheel job uses it too. Might be worth saying so.

Fixed: it says a conda package or a wheel now.

jharlow-intel · 2026-11-09T17:59:50Z

Thanks for going through it. The description is updated: it covers macOS, the wheels, the current pins and the test-job change, and links a run on the current head. On AGENTS.md: mkl_random has none on master yet, as #190 adds it, so whichever of the two lands second will carry the notes for ci/.

Adds a ci/ directory and .github/workflows/intel-ci.yml. The workflow builds conda-recipe/ with conda-build and the wheels with cibuildwheel, for every platform and Python version the recipe and pyproject.toml declare, and tests each package in a clean environment with ci/test.py. conda-recipe/ is now the only recipe, the conda-forge one that was conda-recipe-cf/; the Intel-channel recipe, conda-package.yml and conda-package-cf.yml are removed, as intel-ci.yml builds and tests the same recipe, running the test jobs for each package that built even when another build failed. conda_build_config.yaml now lists the Python versions, and numpy: '2', as conda-build otherwise defaults to 1.26. [build-system] requires mkl-devel, and [tool.gh-ci] pin-compatible maps mkl to it: ci/build.py adds Requires-Dist mkl >= the mkl-devel version python -m build installed, < its next major, the way conda's run_exports pin it, and fails a build that logs no version. It takes METADATA's line ending from the headers. The repair commands exclude MKL's libraries by wildcard, and the Linux wheels are tagged manylinux_2_28 only, like the mkl wheel. The Linux wheels build with the hardening flags conda-forge's compilers set: -fstack-protector-strong, -D_FORTIFY_SOURCE=2 and -Wl,-z,relro -Wl,-z,now. ci/build.py refuses a wheel whose ELF files lack full RELRO. It keeps an RPATH or RUNPATH entry only when it is $ORIGIN or a directory under it, as meson-python keeps the isolated build env's absolute lib directory, and refuses a file left with none. It rewrites the wheels in scratch, so a refused wheel never reaches --output, and refuses two built packages with the same file name, which would overwrite each other there. conda-build gets --extra-meta remote_url=... sha=... when GitHub Actions sets GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_SHA, so each conda package records the commit it was built from in info/about.json. ci/hooks.py tests each wheel in a venv made from conda-forge's Python, as pip users install it. Every python: entry in conda_build_config.yaml names its ABI, which ci/matrix.py requires. ci/matrix.py reads x86 as any x86 CPU and osx64 as undefined, as conda-build does, and refuses a conda-build skip that uses osx64. A pull request also builds and tests the newest free-threaded version, and every push and tag run gets a concurrency group of its own.

jharlow-intel marked this pull request as ready for review October 23, 2026 20:42

jharlow-intel requested review from antonwolfy, ndgrigorian, vlad-perevezentsev and xaleryb as code owners October 23, 2026 20:42

antonwolfy added this to the 1.7.0 release milestone Oct 26, 2026

antonwolfy reviewed Oct 26, 2026

View reviewed changes

jharlow-intel self-assigned this Oct 28, 2026

jharlow-intel requested a review from antonwolfy October 28, 2026 18:31

jharlow-intel mentioned this pull request Oct 28, 2026

Build and test conda packages and wheels IntelPython/mkl-service#265

Merged

jharlow-intel force-pushed the ci/intel-python-ci branch 9 times, most recently from 2ecff7b to 8b5d74a Compare November 3, 2026 18:27

vchamarthi reviewed Nov 4, 2026

View reviewed changes

jharlow-intel changed the title ~~Build and test conda-recipe-cf for Intel Python CI~~ Build and test conda packages and wheels Nov 5, 2026

jharlow-intel force-pushed the ci/intel-python-ci branch 3 times, most recently from cb71ba5 to 75f0985 Compare November 9, 2026 17:59

jharlow-intel force-pushed the ci/intel-python-ci branch from 75f0985 to c7f5e40 Compare November 9, 2026 21:18

	cancel-in-progress: true
	cancel-in-progress: ${{ github.event_name == 'pull_request' }}

		@@ -0,0 +1,135 @@
		# Intel Python CI (gh-ci) for mkl_random: builds the conda-forge recipe, conda-recipe-cf, for every

Conversation

jharlow-intel commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jharlow-intel Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jharlow-intel Oct 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vchamarthi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jharlow-intel commented Nov 9, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jharlow-intel commented Oct 23, 2026 •

edited

Loading

jharlow-intel Oct 28, 2026 •

edited

Loading

jharlow-intel Oct 28, 2026 •

edited

Loading