[vstinner]

• Issue: regrtest: Don't list tests expected to be skipped #161430

[vstinner]

Current status.

Windows:

3 tests skipped:
    test_perf_profiler test_perfmaps test_samply_profiler

macOS:

6 tests skipped:
    test.test_gdb.test_backtrace test.test_gdb.test_cfunction
    test.test_gdb.test_cfunction_full test.test_gdb.test_jit
    test.test_gdb.test_misc test.test_gdb.test_pretty_print

Ubuntu:

1 test skipped:
    test_interpreters

[vstinner]

Emscripten:

200 tests expected to be skipped (not listed)

9 tests skipped:
    test.test_gdb.test_backtrace test.test_gdb.test_cfunction
    test.test_gdb.test_cfunction_full test.test_gdb.test_jit
    test.test_gdb.test_misc test.test_gdb.test_pretty_print
    test__interpreters test_perfmaps test_pty

[vstinner]

iOS:

  123 tests expected to be skipped (not listed)
  
  10 tests skipped:
      test.test_asyncio.test_subprocess test.test_gdb.test_backtrace
      test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
      test.test_gdb.test_jit test.test_gdb.test_misc
      test.test_gdb.test_pretty_print test_asdl_parser test_perfmaps
      test_pty

[vstinner]

Update.

Windows / Build and test (x64, switch-case) and Windows (free-threading) / Build and test (x64, switch-case):

52 tests expected to be skipped (not listed)

1 test skipped:
    test_xxlimited

macOS: 85 tests expected to be skipped (not listed)

macOS (free-threading) / build and test (macos-26): 25 tests expected to be skipped (not listed)

Ubuntu / build and test (ubuntu-26.04): 76 tests expected to be skipped (not listed)

Android (x86_64): 130 tests expected to be skipped (not listed)

iOS: 132 tests expected to be skipped (not listed)

Emscripten: 209 tests expected to be skipped (not listed)

WASI:232 tests expected to be skipped (not listed)

TSan: 63 tests expected to be skipped (not listed)

ASan:

75 tests expected to be skipped (not listed)

27 tests skipped:
    test.test_asyncio.test_subprocess
    test.test_concurrent_futures.test_as_completed
    test.test_concurrent_futures.test_deadlock
    test.test_concurrent_futures.test_future
    test.test_concurrent_futures.test_init
    test.test_concurrent_futures.test_interpreter_pool
    test.test_concurrent_futures.test_process_pool
    test.test_concurrent_futures.test_shutdown
    test.test_concurrent_futures.test_thread_pool
    test.test_concurrent_futures.test_wait
    test.test_multiprocessing_fork.test_manager
    test.test_multiprocessing_fork.test_misc
    test.test_multiprocessing_fork.test_processes
    test.test_multiprocessing_fork.test_threads
    test.test_multiprocessing_forkserver.test_manager
    test.test_multiprocessing_forkserver.test_misc
    test.test_multiprocessing_forkserver.test_processes
    test.test_multiprocessing_forkserver.test_threads
    test.test_multiprocessing_spawn.test_manager
    test.test_multiprocessing_spawn.test_misc
    test.test_multiprocessing_spawn.test_processes
    test.test_multiprocessing_spawn.test_threads test_idle
    test_perf_profiler test_samply_profiler test_tkinter test_ttk

UBSan: 75 tests expected to be skipped (not listed)

MSan:

88 tests expected to be skipped (not listed)

18 tests skipped:
    test.test_concurrent_futures.test_as_completed
    test.test_concurrent_futures.test_deadlock
    test.test_concurrent_futures.test_future
    test.test_concurrent_futures.test_init
    test.test_concurrent_futures.test_interpreter_pool
    test.test_concurrent_futures.test_process_pool
    test.test_concurrent_futures.test_shutdown
    test.test_concurrent_futures.test_thread_pool
    test.test_concurrent_futures.test_wait test_bz2 test_ctypes
    test_dbm_sqlite3 test_gzip test_perf_profiler test_samply_profiler
    test_sqlite3 test_stable_abi_ctypes test_zlib

---

I didn't note tests skipped because of denied resources, such as 3 tests skipped (resource denied): test_peg_generator test_xpickle test_zipfile64 on Ubuntu.

[vstinner]

@cmaloney: Ok, the PR is now ready for review :-)

Lib/test/libregrtest/skipped.py implements multiple different rulesets / heuristics to check if a test is expected to be skipped or not. The rulset is now complete so that there is no more "skipped" tests, only "tests expected to be skipped (not listed)".

The question is now which rulesets/heuristics are a good thing and which ones are a bad thing :-)

I think that the following rules are safe:

• support.MS_WINDOWS
• support.Py_GIL_DISABLED
• sys.platform
• IS_WASI: set from _PYTHON_HOST_PLATFORM environment variable, needed to filter properly tests on WASI

Safe rules which test sys.platform in practice:

• support.is_android
• support.is_apple
• support.is_emscripten
• support.has_socket_support
• support.has_subprocess_support
• support.is_wasm32
• NEED_WORKING_THREADING: IS_WASI and threading_helper.can_start_thread check sys.platform

It should be ok:

• support.has_fork_support: check sys.platform, but check also hasattr(os, "fork"). It should be ok to test hasattr(os, "fork").
• sysconfig.get_config_var('WITH_DTRACE') (./configure --with-dtrace) check for test_dtrace
• sysconfig.is_python_build() check for test_asdl_parser: skip if Python is installed. The test says: "test irrelevant for an installed Python".
• support.supports_perf_trampoline_profiling() for test_perf_profiler, test_perfmaps and test_samply_profiler: check PY_HAVE_PERF_TRAMPOLINE macro which is only set on selected platforms+architectures by configure. So in practice, it checks the platform + architecture.

In the grey area, I'm not sure:

• Test directly hasattr(os, function_name): maybe it should be replaced with a test on sys.platform. Tested functions: startfile() (Windows only), openpty(), wait3() and wait4() (not available on Windows).
• sys.is_remote_debug_enabled(): maybe it should be replaced with a test on sys.platform + platform.architecture(). We mostly care about Py_SUPPORTS_REMOTE_DEBUG macro. If ./configure --without-remote-debug is used, the tests should be reported as skipped. Same if PYTHON_DISABLE_REMOTE_DEBUG env var or -X disable-remote-debug command line option is used.
• shutil.which('gdb') for test_gdb: Should we complain that gdb is not installed, so complain if test_gdb tests are skipped?
• SELECT_TESTS: is it ok to check hasattr(select, name)? devpoll() is available on Solaris, kqueue on FreeBSD and macOS, epoll() on Linux, select() on non-Windows platforms. But having exhaustive list of supported and unsupported platforms is more tricky to write.

Bad idea?

• NEED_MODULES is most risky check. If Python fails to build an extension module (ex: ctypes), NEED_MODULES ignores silently that the test is skipped because of the build error. Maybe it should be replaced with a test on sys.platform. For example, fcntl cannot be built on Windows, so it's safe to expect test_fcntl to be skipped on Windows.

[vstinner]

Sorry, I changed my mind and marked again the PR as a draft. I'm making more changes to address #161432 (comment).

[vstinner]

UPDATE: I updated this comment for my latest commits.

(Yet another) Update.

No tests is skipped on:

• Ubuntu -- same on ARM64 and Free Threading
• Emscripten
• WASI
• iOS
• Lazy imports enabled / Run Tests with lazy_imports=all
• TSan
• UBSan
• Cross build Linux (well, it only runs 3 tests!)

Skip test_gdb, 6 tests skipped: test.test_gdb(...) (all gdb tests):

• macOS: same on ARM64/x86-64 and Free Threading
• Ubuntu (bolt)
• Ubuntu (installed)
• Hypothesis tests on Ubuntu

=> Should we install gdb on these CIs? macOS uses clang and might prefer lldb but we don't provide any helper/script for lldb.

CIs skipping tests:

• Windows / Build and test (x64, tail-call): 2 tests skipped: test_curses test_xxtestfuzz -- same on Free Threading
• Windows / Build and test (x64, switch-case): 3 tests skipped: test_curses test_xxlimited test_xxtestfuzz -- same on Free Threading. Why is test_xxlimited only skipped on switch-case build?
• Android (x86_64): 1 test skipped: test_curses

ASan:

12 tests skipped:
    test.test_asyncio.test_subprocess
    test.test_concurrent_futures.test_as_completed
    test.test_concurrent_futures.test_deadlock
    test.test_concurrent_futures.test_future
    test.test_concurrent_futures.test_init
    test.test_concurrent_futures.test_interpreter_pool
    test.test_concurrent_futures.test_process_pool
    test.test_concurrent_futures.test_shutdown
    test.test_concurrent_futures.test_thread_pool
    test.test_concurrent_futures.test_wait
    test_perf_profiler test_samply_profiler

MSan:

25 tests skipped:
    test.test_concurrent_futures.test_as_completed
    test.test_concurrent_futures.test_deadlock
    test.test_concurrent_futures.test_future
    test.test_concurrent_futures.test_init
    test.test_concurrent_futures.test_interpreter_pool
    test.test_concurrent_futures.test_process_pool
    test.test_concurrent_futures.test_shutdown
    test.test_concurrent_futures.test_thread_pool
    test.test_concurrent_futures.test_wait test_bz2 test_ctypes
    test_dbm_gnu test_dbm_ndbm test_dbm_sqlite3 test_gzip test_lzma
    test_perf_profiler test_readline test_samply_profiler test_smtpnet
    test_sqlite3 test_ssl test_stable_abi_ctypes test_zlib test_zstd

Note: I didn't check "tests skipped (resource denied)": this list is not changed by the PR.

[vstinner]

Ok, new milestone! I modified libregrtest/skipped.py to almost only check sys.platform. I mark the PR ready for review again.

I filtered tests which have dependencies disabled by configure. Example of Apple mobile from configure.ac:

  [iOS|tvOS|visionOS|watchOS], [
    dnl subprocess and multiprocessing are not supported (no fork syscall).
    dnl curses and tkinter user interface are not available.
    dnl gdbm and nis aren't available
    dnl Stub implementations are provided for pwd, grp etc APIs
    PY_STDLIB_MOD_SET_NA(
      [_curses],
      [_curses_panel],
      [_gdbm],
      [_multiprocessing],
      [_posixshmem],
      [_posixsubprocess],
      [_scproxy],
      [_tkinter],
      [grp],
      [nis],
      [readline],
      [pwd],
      [spwd],
      [syslog],
    )
  ],

To make the test report more readable, I also ignored tests with dependencies which are expected to be missing on Emscripten, WASI and Android.

Android:

    # Missing dependencies on Android
    EXPECTED_SKIP |= {
        'test_curses',                 # need _curses
        'test_dbm_gnu',                # need _gdbm
        'test_dbm_ndbm',               # need _dbm
        'test_grp',                    # need grp
        'test_readline',               # need libreadline/libedit
    }

Emscripten:

    # Missing dependencies on Emscripten
    EXPECTED_SKIP |= {
        'test_lzma',                   # need _lzma
        'test_zstd',                   # need _zstd
    }
    # Missing OpenSSL dependency on Emscripten
    EXPECTED_SKIP |= NEED_SSL
    # Missing Tcl/Tk dependency on Emscripten
    EXPECTED_SKIP |= NEED_TKINTER

WASI:

    # Missing dependencies on WASI
    EXPECTED_SKIP |= {
        'test_bz2',                    # need _bz2
        'test_ctypes',                 # need _ctypes
        'test_dbm_sqlite3',            # need _sqlite3
        'test_gzip',                   # need zlib
        'test_lzma',                   # need _lzma
        'test_readline',               # need libreadline/libedit
        'test_sqlite3',                # need _sqlite3
        'test_stable_abi_ctypes',      # need _ctypes
        'test_zlib',                   # need zlib
        'test_zstd',                   # need _zstd
    }
    # Missing OpenSSL dependency on WASI
    EXPECTED_SKIP |= NEED_SSL
    # Missing Tcl/Tk dependency on WASI
    EXPECTED_SKIP |= NEED_TKINTER

@mhsmith @hoodmane @brettcannon: Hi! Are you ok marking these tests as "expected to be skipped" on Android, Emscripten and WASI? Do you see reasons why one of these dependencies may become available in the future? Are you ok to keep these filters for now, and maybe update libregrtest/skipped.py file in the future if a dependency becomes available?

In the worst case, a skipped test is hidden in the final test report, instead of being listed. But the test is still present in full logs obviously, with the reason for the skip.

[hoodmane]

Yeah for Emscripten I'd like to add at least lzma and zstd. OpenSSL is more questionable since it's very big but it can be built. I'd also some day like to make subprocesses work in node. But there's no reason I can't update this new file.

[mhsmith]

This approach adds a maintenance burden, because every time we add, remove or change a skip, we'd now need to update two places.

Maybe we could reduce the log noise by improving the logging instead. Looking at the examples in #161430, by far the largest offender is test_free_threading. On mobile and WASI platforms, test_multiprocessing is also significant. Could we show those packages only once, rather than listing every one of their submodules?

I also notice that skipped modules which are inside packages are shown with a test. prefix (e.g. test.test_asyncio.test_windows_events, which simple modules don't have (e.g. test_msvcrt). Removing this prefix should allow more modules to be listed per line, and reduce the line count further.

[vstinner]

This approach adds a maintenance burden, because every time we add, remove or change a skip, we'd now need to update two places.

If a test is no longer skipped, it will no longer be listed under "200 tests expected to be skipped (not listed)" or under "skipped tests". You don't have to update libregrtest/skipped.py: this file is only to decide if a test fall into "skipped" or "expected to be skipped".

Anyway, I removed the "missing dependencies" rules to focus on the least controversial filters like skipping test_free_threading on a GIL enabled build.

I also notice that skipped modules which are inside packages are shown with a test. prefix (e.g. test.test_asyncio.test_windows_events, which simple modules don't have (e.g. test_msvcrt). Removing this prefix should allow more modules to be listed per line, and reduce the line count further.

Good idea. I also removed test. prefix from TestResults! I'm not sure what adds this test. prefix, but I don't think that anyone needs it.

[vstinner]

Good idea. I also removed test. prefix from TestResults! I'm not sure what adds this test. prefix, but I don't think that anyone needs it.

Oh, because of this change, 145 tests are listed as "omitted" on Ubuntu! display_result() gets again test names with test. prefix for test packages. I wrote #161567 follow-up issue to fix this regression.