Skip to content

[python] New review of the bindings: mechanical fixes - #452

Open
trsxvz wants to merge 17 commits into
masterfrom
444-python-bindings-new-review
Open

trsxvz wants to merge 17 commits into
masterfrom
444-python-bindings-new-review

Conversation

@trsxvz

@trsxvz trsxvz commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

[AI-assisted]

Fixes #444.

Mechanical fixes from the new review of the python bindings. Points which need a decision are listed in a separate comment.

Fixes

Memory errors:

  • integrate and integrate_debug on a BehaviourData check the sizes of its arrays against the given behaviour and hypothesis. A clear error replaces a segmentation fault or a heap overflow. Loading the same behaviour twice still works.
  • The array converted from a buffer object such as array.array now stays alive as long as the manager or the initializer uses it. It is kept by the clean-up hook of [choice] Python bindings: arrays bound as external storage checked and kept alive #377 and by the bind* and useExternalArrayOf* methods. Extending an array.array in use raises BufferError instead of leading to a use after free.
  • An object whose __array__ method returns a new read-only array is refused with NOUPDATE. The manager used to read freed memory.
  • None as self is refused by allocateArrayOf*, releaseArrayOf* and setThreadSafe of MaterialDataManager, instead of a segmentation fault. The change on setThreadSafe is the one made by Add a pip package for the python bindings (issue #254) #449.

Errors and accepted values:

  • The two error_message getters decode with errors="replace", so a truncated or invalid message no longer raises UnicodeDecodeError.
  • setExternalStateVariable on a State accepts all the scalars accepted by the other setters: numpy scalars, Decimal, Fraction, 0D arrays.
  • An empty array given to a bind* method is refused. It used to be silently ignored. This also applies to a behaviour without internal state variables, where binding an empty array was harmless: such a call must now be skipped.
  • A failed import of mfront.database is reported by warnings.warn, and only ImportError is caught.
  • The error message of setExternalStateVariable has its typos fixed.

FEniCS bindings:

  • The KeyError of a missing tangent block names the variable.
  • Numpy scalars are accepted as values of material properties and external state variables.

Documentation and clean-up:

  • The docstring of BehaviourIntegrationResult.n states its value when nothing failed. The value is unchanged.
  • Typos in the docstrings of MaterialDataManager and BehaviourDescription.
  • The std::variant overloads of wrapInNumPyArray, which have no caller, are removed. NumPySupport.hxx is not installed.
  • Entry in the release notes.

A new test, ArgumentsCheckTest, holds a few assertions on the layout check, the kept arrays, the scalars, None as self and the empty arrays.

Not covered

executeInitializeFunction and executePostProcessing only accept a BehaviourDataView in python, and a view does not know its behaviour. These functions and the view overloads of integrate and integrate_debug are left unchanged.

Conflicts with #449

MaterialDataManager.cxx and MaterialStateManager.cxx conflict on the .def of the bind* and useExternalArrayOf* methods: #449 adds named arguments there, this PR removes keep_alive. Keeping the names and dropping keep_alive solves them.

Tested locally

  • ctest -R py: 17/17 with the default build (pybind11 2.11.1), with -Denable-exceptions=ON, and with pybind11 3.1.0.
  • The 17 python tests run under AddressSanitizer: all pass, no report.
  • Reproducers of the review under AddressSanitizer: the 3 cases of a BehaviourData with another behaviour or hypothesis give a RuntimeError, the 4 cases of a reallocated array.array give a BufferError, the new read-only array gives a RuntimeError, the truncated and invalid messages are returned with replacement characters, and the same behaviour loaded twice still integrates.
  • Scalars table of the review: all the scalars are accepted by setExternalStateVariable on a State, by name and by offset.
  • Calls of all the public functions with tuples of valid and invalid arguments: no crash in 1.7 million calls on the default build, and in 60099 calls under AddressSanitizer. These calls crashed on None as self before.
  • Table of 194 reachable errors: same exceptions as before, on the default build and under AddressSanitizer. Only the 2 messages reworded here differ.

Not tested

  • The FEniCS bindings: FEniCS can't run here. The changes were only read and byte-compiled.
  • Windows and macOS.

trsxvz added 16 commits November 7, 2026 01:28
…BehaviourData

The arrays of a BehaviourData were given without check to the behaviour
passed to integrate and integrate_debug. A behaviour needing other arrays,
or the same behaviour for another hypothesis, read and wrote out of bounds:
segmentation fault with Gurson on the data of Elasticity, heap buffer
overflow with Norton in 3D on data in plane strain.

An error is now raised if the sizes of the arrays differ from what the
behaviour and its hypothesis need. The behaviours are not compared, since
loading the same behaviour twice is legitimate.
A buffer object such as an array.array is accepted as external storage, but
the ndarray holding its buffer export was destroyed when the conversion
returned: only the object itself was kept. Extending it then reallocated the
values while the manager still used them, which gave a use after free in the
integration.

The converted array is now what the clean-up hook and the bind and
useExternalArrayOf methods keep. While it is alive, the object refuses to
be resized with a BufferError.
With EXTERNAL_STORAGE and NOUPDATE, a read-only array was assumed to be the
given object or a view of it. An object whose __array__ method returns a new
read-only array breaks this assumption: this array owned the values and was
destroyed at return, so the manager used freed memory.

This case now goes through the check of mgis_convert_to_mutable_span, shared
in a new function. The error message no longer assumes that values are
written.
The buffer of the messages of the behaviours holds 512 bytes: a longer
message may be cut in the middle of a multibyte character, and the message of
a behaviour may not be valid UTF-8. Reading BehaviourData.error_message or
BehaviourIntegrationResult.error_message then raised UnicodeDecodeError, and
the message was lost.
…scalars

Only float and int were treated as scalars. A numpy scalar other than
np.float64, a Decimal, a Fraction or a 0D array went to the path of arrays
and raised "expected one dimensional array", whereas all the other setters
accept them.
…taManager

A member function pointer bound without named argument lets None through as
self, as a null pointer: allocateArrayOfTangentOperatorBlocks,
releaseArrayOfTangentOperatorBlocks, allocateArrayOfSpeedOfSounds,
releaseArrayOfSpeedOfSounds and setThreadSafe ended in a segmentation fault.

These methods have no argument to name, except setThreadSafe: they are now
bound through a lambda taking a reference.
…n success

When nothing failed, n holds the largest value of the size type
(18446744073709551615 on a 64-bit system), which the docstring did not say.
For an initializer, an empty array means that no array is given. An empty
array given to a bind method was thus silently replaced by an internal
storage: the array of the caller was not used, without any error.
"use and externally defined array" and "spped of sounds".
The closing parenthesis was missing in the docstrings of parameters and
integer_parameters, and the latter referred to a property which does not
exist: integer_params instead of iparams.
"setExternalSateVariable" and "1expected", in a message reaching python.
The message of the KeyError was not formatted: it held '{}' instead of the
name of the variable.
A bare except around the import of mfront.database also caught
KeyboardInterrupt and any other error, and the message was printed on the
standard output of every program importing mgis.
type(value) in [int, float] sent np.float64 and the other numpy scalars to
compute_on_quadrature, which failed with an AttributeError on ufl_shape. This
applies to material properties and external state variables.
The three overloads of wrapInNumPyArray taking a std::variant have no caller,
and <iostream> is not used. This header is only used by the sources of the
bindings: it is not installed.
Entry of the issue 444 in the issues fixed by version 3.2.
@trsxvz

trsxvz commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

[AI-assisted]

/!\ not ready to merge yet.

@thelfer the commits of this PR are mechanical. The points below remain open, each with a proposal. "Tried" means that the proposal was built and run on a local copy of this branch, without being committed. The FEniCS proposals could only be read, since FEniCS can't run here.

Two consequences of this PR to check

  • An empty array given to a bind* method is now refused. This also applies to a behaviour without internal state variables, where it was harmless: generic code must skip the call.
  • The converted array now lives as long as the initializer or the manager, through a pybind11::weakref with a clean-up callback, as keep_alive can only hold the arguments of a function. For this, the bind* and useExternalArrayOf* methods now take self as a pybind11::object, as the getters of these classes do. The proposals below on s0 and s1 and on useExternalArrayOf* would make this unnecessary, if MaterialDataManagerInitializer gets the same member.

Crashes and memory errors still reachable

View of a data with another behaviour. The check of this PR applies to a BehaviourData. A view does not know its behaviour: integrate(make_view(d), b) with d built for Elasticity and b for Gurson is still a segmentation fault. executeInitializeFunction crashes in the same way, and executePostProcessing has no check either. Both only accept a view in python, so make_view is the only way to call them on one point. The check only compares sizes: a data of another behaviour with the same sizes is accepted, which is safe for the memory. (S-12)

Proposal: make_view returns a small class, still named BehaviourDataView in python, deriving from the C structure and holding the BehaviourData as a pybind11::object. The 5 overloads taking a view then run the same check. About 30 lines, and the python API does not change. Tried: a RuntimeError in the crashing cases, and the python tests pass. The view keeps its data alive without keep_alive<0, 1>, which also solves the pybind11 3.1.0 point below.

Array bound through s0 or s1 of an initializer. i.s1.bindGradients(numpy.zeros(n)) is accepted, but the array only lives as long as the temporary wrapper i.s1, so until the end of the statement. The manager then uses freed memory: AddressSanitizer reports a use after free in update. The docstring warns the caller, but this is the natural way to write it, and nothing fails at this point. (S-05)

Proposal: add a std::vector<std::shared_ptr<void>> member to MaterialStateManagerInitializer, holding the converted arrays. They then live as long as the C++ initializer, whatever happens to the python wrapper. The warnings in the docstrings of s0 and s1 can be removed. About 15 lines. Tried: the array stays alive with the initializer, then with the manager, and is released with them.

Number of integration points. MaterialDataManager(b, 2**63) with the Elasticity behaviour is accepted: n * stride wraps around to 0 and the arrays are empty. integrate then dereferences a null pointer. Contrived, but it is a segmentation fault. (S-36)

Proposal: check the multiplication in the two constructors of MaterialStateManager. About 5 lines. Tried: a RuntimeError.

Allocation failure. With the default build, a failed allocation while copying an array given with LOCAL_STORAGE, or during update, happens in a MGIS_NOEXCEPT function: std::terminate kills the interpreter. With enable-exceptions the user gets an exception. (S-34)

Proposal: put these constructions of std::vector in setMaterialProperty, setExternalStateVariable and updateMutableFieldHolderFromSpan in a try block calling registerExceptionInErrorBacktrace, as makeExternalStorageHook does. Tried on setExternalStateVariable and update: a RuntimeError holding std::bad_alloc, and the interpreter survives.

make_view(1) with pybind11 3.1.0. A failed conversion with keep_alive<0, 1> gives a segmentation fault instead of a TypeError. This is a defect of pybind11, fixed by pybind/pybind11#6183 on its master, but not in a release yet. make_view is the only binding concerned, and the pip package of #449 asks for pybind11>=2.12, which gives 3.1.0. (S-16)

Proposal: nothing more if the view proposal above is accepted, since keep_alive<0, 1> disappears (read, not run with 3.1.0). Otherwise, exclude this version in pyproject.toml: pybind11>=2.12,!=3.1.0.

Deadlock with pybind11 older than 2.12. The first use of an array wrapped by pybind11 from several python threads can hang: 7 runs out of 8 with pybind11 2.11.1, the version of Ubuntu 24.04. No hang in 10 runs with 3.1.0. CMake accepts any version. (S-32)

Proposal: build one empty pybind11::array_t<double> at the start of PYBIND11_MODULE(behaviour, m), so that the numpy support of pybind11 is initialized during the import. 2 lines. Requiring 2.12 in find_package would refuse the package of Ubuntu 24.04. Tried with 2.11.1: no hang in 10 runs.

Unaligned arrays. numpy.frombuffer(buf, offset=1) is accepted and turned into a std::span<double>. This is undefined behaviour in C++, with no failure seen on x86-64. (S-03)

Proposal: refuse them in mgis_convert_to_span. 3 lines. Tried: a RuntimeError.

Arrays kept for too long

useExternalArrayOf*. These two methods use keep_alive: 50 successive calls keep 50 arrays alive, and releaseArrayOf* releases none, contrary to its docstring. A refused array stays alive too. This is the waste refused in #377 for setMaterialProperty. These methods may be called at each time step. (S-09)

Proposal: they take an optional std::shared_ptr<void> owner, stored by the manager and reset by the release methods, as the hook of #377 does. The views returned by K and speed_of_sound must then hold this owner rather than the manager. Without this, a view taken before a release points to freed memory. About 25 lines. Tried: 1 array alive out of 50, none after the release, and an earlier view stays valid.

Parts of the python API which can't be used

mgis.function. BasicFunction and BasicFunctionView have no constructor, and no function returns them: TypeError: No constructor defined!. The module can't be used. The conversion helper of Function.cxx is never called and returns a type which is not registered. #365 left this open. (S-21, S-22)

Proposal: remove the helper. Add BasicFunction(n, number_of_components) and BasicFunctionView(array, number_of_components), with a check of the sizes in the binding, since a too small array aborts the interpreter with the default build. About 15 lines. Tried: the shapes are right, a view writes in the array, and invalid sizes give a RuntimeError.

mgis.material_property. The module only gives the description of a material property: it can't be evaluated from python. (S-23)

Proposal: add a __call__ method taking a list of values and raising on a negative status. About 10 lines. Tried on a small MFront law: the value is right, and a wrong number of arguments gives the message of the law.

Array without storage mode. setMaterialProperty(s, name, array) raises a TypeError listing 8 overloads, whereas the C++ function defaults to LOCAL_STORAGE. Same for setMassDensity and for setExternalStateVariable on a manager. (S-07)

Proposal: add an overload taking a pybind11::array_t<double> without storage mode, which copies the values. A default value on the existing overload does not suit: it takes a pybind11::object, so it would catch numpy scalars before the overload taking a floating point value. 3 small functions and 6 registrations. Tried on setMaterialProperty: arrays, lists and array.array are accepted, and scalars still work.

Missing bindings. Part of the public C++ API has no binding. There is no way to know from python what a state holds, and python callers compute the sizes of the post-processings by hand. (S-27)

Proposal: bind what has a use in python, all thin wrappers:

  • isMaterialPropertyDefined, isMaterialPropertyUniform, unsetMaterialProperty, the same three for external state variables, isMassDensityDefined, isMassDensityUniform. The unset functions take a Context: the binding turns a failure into an exception.
  • K_stride, getTangentOperatorArraySize, getPostProcessingVariablesArraySize, allocatePostProcessingVariables, getInitializeFunctionVariablesArraySize, allocateInitializeFunctionVariables.
  • integrate(manager, options, dt) without range, update(manager, UpdateArguments), extractInternalStateVariable, ThreadPool.getNumberOfThreads.

Leave out the rest, with no need in python so far: releaseBehaviourIntegrationWorkspaces (256 kB over 200 thread pools), ThreadPool.wait, the setters and getters of a State (the arrays are exposed), convertFiniteStrain* on a BehaviourData, the helpers of Hypothesis, getVariableTypeAsString, print_markdown, the analyser overloads of integrate_debug, which need a C++ class, and the HDF5 save and restore functions, since HDF5 is optional.

FEniCS bindings

Automatic compilation. The bare except: around load_behaviour turns any error, even KeyboardInterrupt, into an attempt to run mfront. For a path such as src/libBehaviour.so, this attempt runs in /. An exception from subprocess.run leaves the current directory changed. (S-28)

Proposal: except RuntimeError:, install_path = os.path.dirname(os.path.dirname(path)) or ".", and subprocess.run([...], cwd=install_path, check=True) instead of os.chdir. About 4 lines.

Integer parameters. update_parameters only calls setParameter: integer and unsigned short parameters are refused. (S-30)

Proposal: choose the setter by name, with iparams and usparams of the behaviour. About 5 lines.

Error message lost. The integration uses the overload returning an int. The message of the behaviour is lost, and the RuntimeError only says "behaviour integration failed". (S-31)

Proposal: use the overload taking BehaviourIntegrationOptions, which returns exit_status, n and error_message. The int overload calls it anyway, so the results do not change. About 5 lines.

Gradients of fixed dimension. Only Tensor and Stensor are converted to vectors. A gradient declared as Stensor3D or Tensor3D raises a TypeError at set-up. (S-29)

Proposal: leave it as it is. It fails with an error and gives no wrong result, and it can't be tested here.

Tests

Python API never run by a test. mgis.material_property, mgis.function, getDatabase, loadFromDatabase, MaterialStateManagerUpdatePolicy, so the read-only path of #377, revert, the rotations, the finite strain conversions, setParameter and its variants, stored_energies, dissipated_energies. The new test only covers part of the initializers and of useExternalArrayOf*. (S-33)

Proposal: a separate PR with one small test file: a few assertions on the read-only path, revert, the rotations and the finite strain conversions. The two modules follow if they get the proposals above.

Proposal: leave these as they are

  • Cycle through C++. An array given with EXTERNAL_STORAGE which is a view of the same manager is never released, nor is the manager. Contrived, and no memory error. (S-08)
  • GIL held during the integrations. Python threads run one at a time, and another thread barely progresses during a long integration. pybind11::call_guard<pybind11::gil_scoped_release> on each overload would solve this, but another thread could then change the arrays during the integration, and no need has been reported. (S-15)
  • FINITESTRAINKINEMATIC_Eto_PK1. Spelled differently from the C++ enumerator. Harmless, and a renaming would break users. (S-19)
  • Stensor3D and Stensor_3D. Variable.getType and the C++ getVariableTypeAsString use two spellings. The C++ function is not bound, so nothing conflicts in python. (S-20)
  • Context of the errors with enable-exceptions. The exception is thrown at the first error, so the context added by the callers is lost. This follows from the design of [choice] enable-exceptions: MGIS_NOEXCEPT, so that errors reported by exceptions reach the caller #399. (S-35)
  • Thread pool and fork. A pool created before a fork has no thread in the child, and integrate hangs. Usual for a thread pool, and python 3.12 already warns about a fork in a multi-threaded process. (S-37)
  • Smaller points. Variable has no __repr__. 2D arrays are refused as external storage, as documented. A view of K taken before a release stays valid but stale. A view has no error_message, but the message can be read on the BehaviourData.

@thelfer do these choices suit, and can they be implemented?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python bindings: new review

1 participant