Repository navigation
Windows 3.14 native parity: test_vector.py process died silently once on main #295
Description
The native parity job on Windows with Python 3.14 failed once on main: the pytest process for tests/test_vector.py died without a traceback, 17 of 46 tests into the file.
- Run: https://github.com/ArcadeData/arcadedb-python/actions/runs/42187538256 (the push of the Docs: Python functions and record events, with tests that pin each statement first (refs #267) #284 merge commit
361e36efa3), jobParity against the installed wheel (windows-x64, Python 3.14). - The per-file log in the uploaded
native-parity-windows-x64-py3.14artifact showscollected 46 itemsand then 17 passing dots and nothing else: no failure, no Python traceback, noWindows fatal exceptiontext. No XML was written, so the runner reportedtest_vector.py(CRASH)with 0 tests, while the other files of that job passed (2432 passed, 0 failed). - Counting the tests in file order, the 18th is
TestLSMVectorIndex::test_lsm_vector_search_approximate_returns_k(a count ofdef test_in the file at that commit, not a confirmed trace). - It does not reproduce: the PR run of the same code was green on Windows, and the next push to
main(Return a Python str from Document.get_type_name() on arcadedb-jvm #287,f0d594a124) was green on every platform. Docs: Python functions and record events, with tests that pin each statement first (refs #267) #284 only added docs and tests. native/wheel/parity_wheel.pyalready carries a comment that whole-file runs oftest_vector.pyon Windows failed in the past, when the engine logged to the console under pytest's fd capture. This may be the same family, but nothing in this log shows it.
What is missing is the exit code of the dead process. An access violation, a kill by the runner and a pytest-internal abort all look the same in the current log. I am sending a one-line change so the next occurrence records it (CRASH(rc=...) in the status and the log). I will not retry crashed files automatically: one occurrence does not justify hiding a native crash.
If it happens again on Windows, this issue collects the occurrences.
Activity
tae898 commentedon Nov 10, 2026
Two more occurrences, both in the release run of
26.11.1.post1(run 42744219731, tag commitdbe16709dd), jobParity against the installed wheel (windows-x64, ...):tests/test_free_threading.pycollected 4 items, then nothing: no dot, no traceback, no XMLtests/test_find_nearest_query_forms.pyPython 3.12 and 3.13 on Windows passed in the same run. The gate reports
CRASHwithout a return code: the one-line change that recordsCRASH(rc=...)is not inrelease/26.11.1, so the exit code of the dead process is still unknown. Counting Windows parity legs of this code: the two full-matrix dispatches on the post1 candidate (3.12 and 3.14 each, 4 legs) had no crash; this release run has 2 crashes in 4 legs. The first occurrence (this issue) was onmainbefore any post1 change, so the crash is not new in post1, but I cannot say yet whether its rate changed.Next: re-run the failed legs of the release run; if the native wheel is published I will add a Windows-only loop (the two files, 30 repetitions per Python version, the 26.11.1 wheel against the post1 wheel) to estimate the rates, with the return code recorded.
tae898 commentedon Nov 10, 2026
The control run of the release configuration finished (run 42758616458): tag checkout (
dbe16709dd), the stock harness (faulthandler disabled as in the release, no diagnostics), the wheel of the release run, Python 3.11 and 3.14, 5 repetitions each. All 10 legs ran all 148 test files to the end: 2,911 passed, 0 failed, 0 errors, 105 not applicable on the native wheel, no crash and no timeout. The jobs show red only because the throwaway workflow's own summary step exited 2; the parity step passed in each.So the stock harness configuration does not by itself explain the crash: it did not happen in these 10 legs, and it did not happen in the 18 legs with diagnostics or in the 360 isolated runs of the two files either, while the release run lost a file in its Python 3.11 and 3.14 legs in both attempts. I do not know the cause. Something differs between the release run and every run I can start, and the next thing I can do is make the gate record it.
For
26.11.1.post2I am changing the Windows parity harness only (no wheel code): recordCRASH(rc=0x...), keep faulthandler output in a file for the native run and upload it with the parity artifact, retry a crashed file once in a fresh process on Windows only, mark a passing retryFLAKY-CRASHin the job summary and comment it here, and fail the leg and block the release when a file crashes twice. Test failures stay strict, and so do Linux and macOS. Tracking is in #347. If you have a Windows machine with Python 3.11 or 3.14 and want to help, I can share the wheel of the release run and a loop script that runs the two files withpython -X faulthandler.tae898 commentedon Nov 10, 2026
The Windows parity harness of the native release now handles the crashes this issue collects (in the tag
26.11.1.post2, CI and tests only):CRASH(rc=0x...), with the exit code of the process;faulthandlerwrites to a file, and the file is uploaded with the parity artifact, so the crash site is kept;FLAKY-CRASHin the job summary, and thecrash-reportjob oftest-native.ymlposts it here; a file that dies twice fails the leg and blocks the release;The full-matrix dispatch of the native workflow on the post2 commit (run 42789425439) had no crash event: the
Report parity crashesjob printedno crash event in this run. The same holds for the release run of the tag (run 42804220685): the four Windows parity legs (Python 3.11 to 3.14) passed, andReport parity crashesprintedno crash event in this run. The harness has not yet met a real crash.Still unknown: why the processes of the
26.11.1.post1release run died, in two files on two Python versions in both attempts, while 10 control legs, 18 diagnostic legs and the isolated runs did not reproduce it. The exit code and the fault trace are the missing facts, and the next occurrence records them.I keep this issue open and append every
CRASHandFLAKY-CRASHwith its exit code and trace. A clean release run does not close it: the first occurrence was a single one as well.CI: Windows parity harness records crash exit codes, keeps a trace an…
integration: merge ci/post2-windows-crash-retry (aa223d0: Windows-onl…
integration: merge ci/post2-windows-crash-retry again (fc7b5e2: DIRTY…
Merge the 26.11.1.post2 release line into main
tae898 commentedon Nov 12, 2026
Another occurrence on Windows, and this time the crash has a frame. In the examples job of the run for #374,
11_vector_index_build.py(vector build,arcadedb_sqlbackend, minimal) died 7 s in withEXCEPTION_ACCESS_VIOLATION (0xc0000005)in a C2-compiled jvector method,io.github.jbellis.jvector.graph.diversity.VamanaDiversityProvider.isDiverse. The JVM was Corretto 25.0.5+10 on windows-amd64 (G1, compact object headers). The same job passed in the three latest runs on main and in a rerun of the failed jobs, so it is intermittent.The hs_err log and the minidump are kept as the artifacts
jvm-crash-logdir-windows-amd64-py3.14andjvm-crash-cwd-windows-amd64-py3.14of https://github.com/ArcadeData/arcadedb-python/actions/runs/43137532222. I have not checked yet whether jvector's native or Vector API path is involved on Windows. If it happens again I will file it with the jvector project with the hs_err attached.