Skip to content

Add a pip package for the python bindings (issue #254) - #449

Open
thelfer wants to merge 14 commits into
masterfrom
254-add-a-pip-package
Open

thelfer wants to merge 14 commits into
masterfrom
254-add-a-pip-package

Conversation

@thelfer

@thelfer thelfer commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

Fixes #254.
Fixes #319.

Add a pip package for MGIS: Python bindings only, with a cibuildwheel job modeled on the tfel-mfront wheel.

Commits

  1. cmake: options to mandate TFEL and to install the python modules elsewhere — MGIS_PYTHON_MODULES_INSTALL_DIRECTORY can be set by the user (the wheel passes site-packages), as TFEL does with TFEL_PYTHON_SITE_PACKAGES_DIR. require-tfel makes the configuration fail when TFEL is not found, when it provides no MFront support, when the python bindings of TFEL are missing, or when the TFEL version differs from MGIS_REQUIRED_TFEL_VERSION.
  2. cmake: generation of the stubs of the python modules with pybind11-stubgen — mirrors the reviewed TFEL design: pybind11-stubgen is looked for and run with Python_EXECUTABLE; the modules are placed next to the __init__.py script of their package in the build tree (generator expression, so that multi-configuration generators do not append the configuration name); sitecustomize.py-based DLL registration on Windows; py.typed installed.
  3. pip: packaging of the mgis wheel — bindings/python/pip/{pyproject.toml,setup.py,test_wheel.py}. The mgis module is the MGISHOME directory (bin, lib, include, share) with the python modules at its root.
  4. ci: job building and publishing the wheels — cp39–cp315 on linux x86_64, macOS arm64 and windows AMD64, cp312 only on pull requests; wheels published to PyPI and attached to the GitHub release on MFrontGenericInterfaceSupport-* tags.
  5. docs: installation of the python bindings using pip.

Consistency with the TFEL wheel

The wheel is built against, and depends on, tfel-mfront==5.2.0.dev0 — one constant in setup.py, mirrored twice in pyproject.toml and checked at build time. Five layers make an inconsistent pair of wheels impossible:

  1. cmake refuses to configure against another TFEL (require-tfel, MGIS_REQUIRED_TFEL_VERSION),
  2. the runtime paths of all the binaries of the wheel point to the sibling tfel/lib directory (set as DT_RPATH, so LD_LIBRARY_PATH cannot take precedence),
  3. auditwheel/delocate are prevented from vendoring the TFEL libraries, which would make a behaviour load two copies of TFEL (exclusion patterns are globs since auditwheel 6; the old regex-style ones silently match nothing),
  4. importing mgis checks the version of the installed tfel-mfront against the baked one and refuses with an explicit message,
  5. test_wheel.py runs in the fresh environment created by cibuildwheel: it compiles a behaviour with the mfront of the TFEL wheel, integrates it through mgis.behaviour and checks the analytic result, and builds a consumer project using find_package(MFrontGenericInterface) against the wheel.

Local validation

tfel-mfront is not on PyPI yet (thelfer/tfel#1083 is still open), so the wheel was built and tested against a simulated tfel-mfront installation, then installed in fresh virtual environments and tested with and without auditwheel repair, with LD_LIBRARY_PATH, PYTHONPATH and TFELHOME unset. Consequences:

  • the wheel job of this PR fails until tfel-mfront is published: pip cannot resolve the pinned build dependency in the cibuildwheel containers.

Additions [AI-assisted]

Commits added on top of the five above, one per point:

  • cmake: the generation of the stubs extends PYTHONPATH instead of replacing it: pip wheel from the sources stopped with No module named pybind11_stubgen.
  • cmake: install the py.typed marker only if the stubs are generated: without stubs, mypy reported the modules as not found.
  • python: python names of the types in the signatures of the behaviour and model modules: the stubs had ... or a type which does not exist (s0, s1, make_view, mgis.model.load).
  • python: names of the arguments of the behaviour module, and of the material_property and model modules: no plain method shows arg0 anymore. The special methods and the setters of the properties are left as they are, as in 1072 add support for pybind11 stubgen tfel#1083.
  • pip: the test of the wheels checks that the stubs are valid python, as in 1072 add support for pybind11 stubgen tfel#1083.
  • pip: the libraries of the tfel-mfront wheel are not versioned: the comment described a versioned soname, whereas this wheel uses -Dunversioned-libraries=ON.
  • remove the design note of the pip package: pip-package.md was a proposal written before the implementation. It remains in the history of this branch (0fd9d87).
  • docs: release notes of the pip package and of the stubs.

Tested locally, on Linux, against a real tfel-mfront wheel built with pip wheel from thelfer/tfel#1083, and not a simulated one:

  • pip wheel builds the mgis wheel. Installed in a new environment with the tfel-mfront wheel, test_wheel.py passes, raw and after auditwheel repair.
  • Each TFEL library is loaded once. The repair of the tfel-mfront wheel creates no .libs directory, so it does not ship the TFEL libraries twice.
  • mypy gives no error on the imports of mgis, mgis.behaviour, mgis.material_property, mgis.model and mgis.function.
  • ctest -R py: 16/16, on a regular build against an installed TFEL.

Still open

  • The wheel job stays red until tfel-mfront is published, as stated above.
  • The wheel is named mgis-3.1.0.dev0, since CMakeLists.txt has VERSION 3.1.0, whereas the tag of 3.1 exists and the release notes in progress are those of 3.2.
  • Windows and macOS were not tested.

…where

Two changes required by the pip package (issue #254):

- MGIS_PYTHON_MODULES_INSTALL_DIRECTORY can now be set by the user
  (the wheel passes 'site-packages'), as TFEL does with
  TFEL_PYTHON_SITE_PACKAGES_DIR.
- require-tfel makes the configuration fail when TFEL is not found,
  when it provides no MFront support, when the python bindings of
  TFEL are missing while those of MGIS are enabled, or when the TFEL
  version differs from MGIS_REQUIRED_TFEL_VERSION. The wheel build
  passes both so that it can never be produced against another TFEL
  than the one it depends on.
…ubgen

The stubs (.pyi files) of the python modules are generated by
pybind11-stubgen and installed next to the modules, when the tool is
available for the python interpreter of the bindings. The
enable-pybind11-stubgen option turns this off. An empty py.typed
marker is installed in the package (PEP 561).

This mirrors the design of TFEL (thelfer/tfel#1083, reviewed):

- pybind11-stubgen is looked for and run with Python_EXECUTABLE, as
  it imports the modules it describes: the tool must be available for
  the interpreter the modules are built for. Nothing is generated,
  and nothing fails, when it is not available.
- the modules of the package mgis are now built next to its
  __init__.py script in the build tree, which multi-configuration
  generators prevent by appending the configuration name to the
  output directory: it now holds a generator expression. The python
  tests, which assemble the package by symlinks, are unaffected.
- on Windows, a sitecustomize script, only used when the stubs are
  generated, adds the directories of the dynamic link libraries of
  the build tree, given by the new MGIS_LIBRARIES global property,
  and the bin directory of TFELHOME to the search path of the
  dynamic loader. os.add_dll_directory is only called on directories
  which exist.

cf. issue #254
The mgis package (issue #254): setup.py configures, builds and installs
MGIS with cmake, and lays the installation out as a python package, the
mgis module being the MGISHOME directory (bin, lib, include and share)
with the python modules at its root.

The wheel depends on the tfel-mfront wheel, pinned both as a build and
as a run dependency, the pin being written in three places which
check_tfel_dependency verifies consistent:

- cmake is passed require-tfel and MGIS_REQUIRED_TFEL_VERSION, so that
  the wheel can never be built against another TFEL than the pinned one,
- the runtime paths of all the binaries of the wheel point to mgis/lib
  and to the tfel/lib directory of the sibling wheel, set as DT_RPATH so
  that LD_LIBRARY_PATH does not take precedence over them,
- the version of the pinned tfel-mfront is recorded in the __init__.py
  script of the package, which refuses to import when the installed
  version differs, as the error of the dynamic loader would be cryptic,
- auditwheel and delocate are prevented from vendoring the TFEL
  libraries into this wheel (pyproject.toml), which would make a
  behaviour loaded in python load two copies of TFEL. test_wheel.py
  checks that the wheel holds none of them.

test_wheel.py, which cibuildwheel runs in a fresh virtual environment,
compiles a behaviour with the mfront of the tfel-mfront wheel,
integrates it through mgis.behaviour and builds a consumer project using
find_package(MFrontGenericInterface) against the wheel.

The first version only supports the python bindings, on linux x86_64,
macOS arm64 and windows AMD64.
cibuildwheel builds the wheels of the python bindings for cp39 to cp315
on linux x86_64, macOS arm64 and windows AMD64 (configuration in
bindings/python/pip/pyproject.toml), and runs test_wheel.py on them.
Only one python version, cp312, is built for the pull requests.

The compilations of the linux and macOS wheels use the ccache actions of
the repository, shared by the versions of python through the key
wheel-<os>. The Windows build is not accelerated: the ccache actions
only support Linux and macOS.

The wheels are published to PyPI and attached to the GitHub release when
a MFrontGenericInterfaceSupport tag is pushed, which also requires the
tfel-mfront wheel the package depends on to be published (cf. issue #254).
@thelfer thelfer linked an issue Nov 6, 2026 that may be closed by this pull request
…acing it

pybind11-stubgen and the python bindings of TFEL may only be found through
this variable, as in the isolated build environments of pip: pip wheel and
pip install from the sources stopped with "No module named pybind11_stubgen".
Without stubs, this marker makes mypy report the modules as not found, as
pybind11 modules do not provide any type information by themselves.
…and model modules

A type declared after a function using it appears by its C++ name in the
signature of this function. pybind11-stubgen can not handle such a name, which
left "..." or a type which does not exist in the stubs: s0 and s1 of
MaterialDataManager and of its initializer, make_view, and the functions of
the model module.
The stubs generated by pybind11-stubgen showed these arguments as arg0, arg1,
and they can now be passed by keyword. The special methods and the setters of
the properties are left as they are.
It was a proposal, written before the implementation. It remains available
in the history of this branch (0fd9d87).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for pybind11_stubgen Add a pip package

2 participants