Repository navigation
Build and test conda packages and wheels - #2409
jharlow-intel wants to merge 1 commit into
Conversation
9fc1cc4 to
e53ef4e
Compare
|
View rendered docs @ https://intelpython.github.io/dpctl/pulls/2409/index.html |
|
coverage: 75.173%. remained the same — ci/intel-python-ci into master |
| if os.environ.get("GH_CI_KIND") == "wheel": | ||
| print("deselecting test_syclinterface: a wheel keeps no symlinks") | ||
| skipped.append("test_syclinterface") | ||
| if not has_compilers(env): |
There was a problem hiding this comment.
If the cell's config lists compilers, a failed probe should fail the run rather than deselect test_c_headers/test_cython_api. As is, a broken toolchain still passes.
There was a problem hiding this comment.
Done: when the test environment has conda's compiler packages, as the conda runs do from ci/config.yml, a failed probe now fails the run. The wheel environments have only the system's compilers, so there it still deselects the two tests and says so.
| gpu = os.environ.get("GH_CI_DEVICE") == "gpu" | ||
| with tempfile.TemporaryDirectory() as merged: | ||
| env = sycl_env(gpu, pathlib.Path(merged)) | ||
| subprocess.run([sys.executable, "-m", "dpctl", "-f"], env=env) |
There was a problem hiding this comment.
The return code is ignored. Checking it would report an import or runtime failure here instead of later as a missing device.
There was a problem hiding this comment.
Fixed: a non-zero exit from python -m dpctl -f now fails the run with that code, before the device check.
| run: | | ||
| cd "$PACKAGES" | ||
| conda_packages=(dpctl-*.conda) | ||
| version=${conda_packages[0]#dpctl-} |
There was a problem hiding this comment.
The wheels are uploaded with the conda package's version, which can differ from the wheel's own PEP 440 version. If no dpctl-*.conda matches, the unexpanded glob ends up in version. Taking the version from each wheel, or at least failing on an empty glob, would avoid both.
There was a problem hiding this comment.
Done: each wheel is uploaded with its own version, taken from its file name (PEP 427), and the step fails with an error when there's no conda package or wheel to upload, instead of passing the glob on.
| # with the run rather than with the machine's user. | ||
| home = pathlib.Path.cwd() / "home" | ||
| home.mkdir(exist_ok=True) | ||
| env = dict(os.environ, HOME=str(home)) |
There was a problem hiding this comment.
On Windows ~ resolves from USERPROFILE, not HOME, so .asv-machine.json still goes to the agent's profile. Setting USERPROFILE as well would fix it.
There was a problem hiding this comment.
Right, fixed: it sets USERPROFILE as well as HOME, so ~ resolves to the scratch home on Windows too.
| if DEFINED OVERRIDE_INTEL_IPO ( | ||
| set "CMAKE_ARGS=%CMAKE_ARGS% -DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=FALSE" | ||
| REM useful for building in resources constrained VMs (public CI) | ||
| set "CMAKE_ARGS=%CMAKE_ARGS% -DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=FALSE" |
There was a problem hiding this comment.
The Intel-channel recipe only did this under OVERRIDE_INTEL_IPO. It is now unconditional, so release Windows packages are built without IPO, and build.sh has no equivalent. Is that intended?
There was a problem hiding this comment.
Yes, that's intended: it matches conda-forge's dpctl-feedstock (recipe/build.bat) and the conda-recipe-cf/bld.bat this recipe replaces, which both turn IPO off on Windows unconditionally; the feedstock's build.sh has no equivalent either. conda-package.yml also set OVERRIDE_INTEL_IPO=1 for its Windows builds, so the dppy/label/dev packages were already built without IPO.
| :: Make CMake verbose | ||
| set "VERBOSE=1" | ||
|
|
||
| :: set CMAKE to use less threads to avoid OOM |
There was a problem hiding this comment.
The comment says fewer threads, but CPU_COUNT is the full count, so this doesn't limit anything. Same in build.sh.
There was a problem hiding this comment.
Fixed the comment in both files: it now says it runs one build job per CPU that conda-build counts, and that a lower CPU_COUNT saves memory. The line itself stays as the feedstock has it.
| ] | ||
| environment = { CC = "/opt/intel/oneapi/compiler/latest/bin/icx", CXX = "/opt/intel/oneapi/compiler/latest/bin/icpx", CMAKE_GENERATOR = "Ninja", CMAKE_ARGS = "-DDPCTL_LEVEL_ZERO_INCLUDE_DIR=/opt/level-zero/include -DDPCTL_WITH_REDIST=ON" } | ||
| # The SYCL and icx runtime libraries come from wheels at run time, see [tool.gh-ci] | ||
| repair-wheel-command = "auditwheel repair --exclude libsycl.so.9 --exclude libintlc.so.5 --exclude libimf.so --exclude libsvml.so --exclude libirng.so -w {dest_dir} {wheel}" |
There was a problem hiding this comment.
libsycl.so.9 here (and sycl9.dll in the delvewheel list) is pinned, while wheel-requires allows any 2026.x runtime. If the soname changes, the repair step vendors a second libsycl. Worth a comment tying these to wheel-requires.
There was a problem hiding this comment.
Done: the repair commands now match any libsycl.so.* and sycl*.dll, with a comment tying them to wheel-requires, so a new SYCL runtime soname is still left out of the wheel.
jharlow-intel
changed the title
a7d1572 to
9ea68d5
Compare
Adds a ci/ directory and .github/workflows/intel-ci.yml, which build the conda packages and the wheels for every declared platform and Python and test each with ci/test.py on the OpenCL CPU device. conda-recipe/ is now the only recipe, the conda-forge one that was conda-recipe-cf/; the Intel-channel recipe and conda-package.yml are removed. The dev uploads to anaconda.org and their cleanup move into intel-ci.yml, for pushes to master and tags only. The wheels leave the SYCL runtime out and require intel-sycl-rt and intel-cmplr-lib-rt, as the PyPI wheels do; scikit-build is pinned below 1.0 (#2341). LDFLAGS in the cibuildwheel environment links every Linux library with -z,relro,-z,now, as conda-forge's compiler activation does for the conda packages, so libDPCTLSyclInterface.so gets full RELRO like the extensions. conda-build gets --extra-meta remote_url=... sha=... when GitHub Actions sets GITHUB_SERVER_URL, GITHUB_REPOSITORY and GITHUB_SHA, so each conda package records the commit it was built from. A pull request also builds and tests the newest free-threaded version, and every push and tag run gets a concurrency group of its own. The benchmarks take PyYAML from conda-forge, as PyPI has no Python 3.15 wheel for it yet. ci/build.py refuses two built packages with the same file name, which would overwrite each other in its flat output directory. ci/test.py fails when python -m dpctl -f fails, and when the environment's conda compilers cannot build an extension, instead of deselecting the two tests that need them. The dev upload takes each wheel's version from its file name and fails when there is nothing to upload. The benchmarks set USERPROFILE as well as HOME, and the repair commands match any libsycl version, as wheel-requires allows any 2026 runtime.
9ea68d5 to
a3fdfd1
Compare
Adds
ci/and.github/workflows/intel-ci.yml, which build the conda packages and wheels for Linux and Windows on every declared Python, and test each in a clean environment.conda-recipe/is now the only recipe: the conda-forge one that wasconda-recipe-cf/. The Intel-channel recipe andconda-package.ymlare removed;conda-package-cf.ymlstays.dppy/label/devuploads and the cleanup of old dev packages move intointel-ci.yml, for pushes tomasterand tags only.intel-sycl-rtandintel-cmplr-lib-rt, as the PyPI wheels do.scikit-buildis pinned below 1.0 (Scikit-build (classic) 1.0 internals #2341).ci/test.pyrunspytest --pyargs dpctlon the installed package and fails when it finds no SYCL device: the OpenCL CPU device in Actions, or a GPU withGH_CI_DEVICE=gpu.Validation, with a temporary push trigger on this branch: run 41648323136 built and tested the conda packages and wheels for 3.10 and 3.15 on Linux and Windows, and pre-commit, zizmor and
conda-package-cf.ymlpassed. Before the rename, a full-matrix run passed all 61 jobs, 3.14t and 3.15t included. Not run yet: a GPU, and the uploads, which onlymasterand tags do.