Skip to content

feat: attach all threads to app-hang events - #2327

Open
tustanivsky wants to merge 22 commits into
masterfrom
feat/app-hang-all-threads
Open

tustanivsky wants to merge 22 commits into
masterfrom
feat/app-hang-all-threads

Conversation

@tustanivsky

@tustanivsky tustanivsky commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Adds sentry_options_set_app_hang_thread_stackwalk_mode (experimental), so app-hang events can also carry the stack traces of all other threads. The event then shows what the hung thread waits on, for example a worker that holds a lock or the render thread.

Key changes

  • New option sentry_options_set_app_hang_thread_stackwalk_mode, which reuses sentry_thread_stackwalk_mode_t. It defaults to the new SENTRY_THREAD_STACKWALK_MODE_HUNG_ONLY alias of CRASHED_ONLY, which keeps the current app-hang event. ALL attaches all threads.
  • The hung thread is added as crashed without a stack of its own. Its stack stays on the exception, which links to it through thread_id. Other threads are added with their names and stacks.
  • At most 256 threads and 1 s of walking per event. Threads other than the hung one get at most 100 ms each.
  • Thread listing on Windows x64 (ToolHelp32), Linux and Android (/proc/self/task) and macOS (task_threads). On Xbox and Windows x86/ARM64 the option is ignored with a warning.
  • Linux and Android: each sample uses its own slot, so a signal handler that runs late cannot mix its stack into another sample. Threads that still have a pending sample signal are skipped.

Known limitations

These apply only with ALL. HUNG_ONLY keeps the current behavior.

  • Windows: threads are suspended and walked with dbghelp. A thread suspended while it holds a lock the walk needs, such as the heap lock, can block the watchdog. This was already possible for the hung thread; ALL extends it to every thread. A stress run with threads loading modules, allocating and calling dbghelp did not hang.
  • Android: each thread is parked in the signal handler while the watchdog walks its stack with libunwindstack, which allocates. A thread parked while it holds the heap lock blocks the watchdog. A run on a Galaxy S23 walked 76 of 78 threads in 331 ms without a hang.
  • Linux and Android: the stack is walked in a signal handler that runs on each thread. A thread with a very small stack can overflow it.
  • Linux and Android: the signal interrupts blocking calls such as nanosleep, poll or epoll_wait, which can return early with EINTR in code that does not retry them.

Related items

Add sentry_options_set_app_hang_attach_all_threads (experimental, off by
default). When enabled, the app-hang watchdog lists and walks every other
thread and adds it to the event. The hung thread is marked as crashed and
linked to the exception through thread_id. The walk is limited to 256
threads and a 1 s budget.

Windows x64 now unwinds suspended threads without dbghelp and without
RtlLookupFunctionEntry, which takes the loader function table lock that a
suspended thread can hold. Linux ignores late signals from timed-out samples
that arrive while another thread is sampled. Logging no longer happens while
a thread is suspended.
Add app_hang_all_threads_during_exceptions: one thread keeps throwing and
catching C++ exceptions, another keeps allocating and freeing, while the
watchdog captures 50 hangs with all threads attached.

On macOS the module loading stress test opened libz.so.1, which does not
exist there, so dyld never loaded or unloaded an image. It now loads a test
module built outside the dyld shared cache, so that dlclose() unmaps it.
Guard the ToolHelp32 thread listing for Xbox, where it is not available.
Read thread names the same way as the native crash daemon. Reuse the hung
thread's id for the exception's thread_id.

Keep two tests: one checks the event shape with the real stackwalker, the
other is the regression test for the Windows loader lock deadlock. Remove
the remaining stress tests, the macOS test module and the pytest test.
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Messages
📖 Do not forget to update Sentry-docs with your feature once the pull request gets approved.

Generated by 🚫 dangerJS against 4d8c90b

Comment thread tests/unit/test_app_hang.c Resolved
Comment thread tests/unit/test_app_hang.c Resolved
Skip both tests under qemu-user, which does not deliver thread-targeted
signals faithfully. Run the module loading test only on Windows and Linux,
where the loaded module really loads and unloads. Other platforms have no
dlfcn.h (PlayStation) or keep libz loaded.
Comment thread tests/unit/test_app_hang.c Resolved
Windows x86, ARM64 and MinGW still unwind with dbghelp, which can block on
the loader lock that the suspended thread holds. Move the unwinder switch
to the stackwalk header so the test can use the same condition.
Linux and Android skip threads that block the sampling signal, and wait at
most 100 ms for other threads instead of 1 s, so one thread that does not
respond cannot use up the walk budget. The hung thread keeps the 1 s wait.

Attaching all threads now needs the lock-free unwinder on Windows. Builds
that still unwind with dbghelp (x86, ARM64, MinGW) ignore the option and
log a warning, like Xbox.
A handler that already passed its checks can still run after its sample
timed out. It writes the shared sample state, so the next sample could
overlap with it. With the shorter wait for other threads this happened on
Arm64 under TSan. Count running handlers and let a new sample wait until
none are left.
Comment thread src/sentry_thread_stackwalk_posix.c Resolved
is_signal_blocked runs before every sample, including the hung thread's,
so read /proc/self/task/<tid>/status with open/read into a stack buffer
instead of fopen, which allocates.
Comment thread src/sentry_thread_stackwalk_posix.c Resolved
…on Windows only

The wait for a late handler now counts against the sample's timeout, so a
sample no longer waits up to twice as long.

The module loading test covers the Windows loader lock. On Linux it only
added dlopen traffic, and under TSan on Arm64 the test process died
intermittently without a report.
The watchdog sends the signal with rt_tgsigqueueinfo and the address of a
per-sample slot, as Unreal Engine does for its own thread stack walks. The
handler writes only to that slot, so a handler that runs after its sample
timed out can no longer mix its stack into the next sample. A slot is
reused only when no handler runs in it, and only the first handler on the
target thread claims a sample.

This replaces the shared sample buffers, the target check and the count of
running handlers.
Comment thread src/sentry_thread_stackwalk_posix.c Resolved
The handler checks the target thread before it claims a sample, so it has
to read the target with acquire to see the slot fields the watchdog wrote
before it. The watchdog still sets active last, so a late handler for the
slot's previous thread cannot claim the new sample.
The slot now stores the id of the thread whose handler may claim it. The
handler checks the id and claims the sample with a single
compare-and-swap, which always works on the latest value, so it can no
longer read a stale target before the claim.
Valgrind blocks all signals of every thread in the kernel, so the SigBlk
check skipped every thread, including the hung one, and no hang event was
created. Skip a thread only when the signal of an earlier sample is still
pending on it. A thread that blocks the signal then costs one short wait in
its first sample, and no further signals queue on it.

The handler now finds its slot by thread id instead of a value carried with
the signal, so the sample is sent with tgkill again.
The module finder reads the headers of every loaded module when it builds
the list for the first event. The loader thread could unload winmm.dll
between the snapshot and the read, which crashed the test on Windows.
…rind

Valgrind runs one thread at a time and delivers signals late, so the test
runner thread can miss the short wait for threads other than the hung one.
The test still runs under valgrind to check for memory errors.
getenv was used without stdlib.h, which broke the PlayStation build.
Valgrind only runs on Linux, so the check and its includes are Linux only.
clang-cl builds with -Werror and reported -Wcast-align on the casts from
the image base to the PE headers and the function table, and
-Wlanguage-extension-token on __try. Cast through const void * and silence
the extension warning around the SEH block, as sentry_sync.c does.
@codecov

codecov Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.18121% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.77%. Comparing base (cbc4604) to head (4d8c90b).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2327      +/-   ##
==========================================
+ Coverage   75.74%   75.77%   +0.03%     
==========================================
  Files         104      104              
  Lines       28443    28706     +263     
  Branches     5220     5265      +45     
==========================================
+ Hits        21545    21753     +208     
- Misses       5543     5585      +42     
- Partials     1355     1368      +13     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Restore the original dbghelp walk on Windows. A stress run with all
threads walked while another thread loaded and unloaded modules, allocated
and called dbghelp did not hang, so the custom unwinder is not needed. This
also keeps walking through JIT code registered with RtlAddFunctionTable.

Attaching all threads stays limited to Windows x64, where the cross-thread
dbghelp walk is validated, and is off on Xbox and ARM64EC. Remove the module
loading test, the moved debug logs and the reworked qemu skip list.

@jpnurmi jpnurmi left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to reuse the existing sentry_thread_stackwalk_mode_t instead of a bool? Something like:

sentry_options_set_app_hang_thread_stackwalk_mode(opts, SENTRY_THREAD_STACKWALK_MODE_ALL);

defaulting to SENTRY_THREAD_STACKWALK_MODE_CRASHED_ONLY. It's the same concept with the same two values, the hung thread is already reported as crashed, and it keeps one vocabulary across crash and app-hang reporting, with room for more modes later.

EDIT: An alias is also a possibility if CRASHED_ONLY sounds misleading...

Replace sentry_options_set_app_hang_attach_all_threads with
sentry_options_set_app_hang_thread_stackwalk_mode, which takes the existing
sentry_thread_stackwalk_mode_t. CRASHED_ONLY is the default and keeps the
current app-hang event, with only the hung thread. ALL attaches the stack
traces of all threads.
SENTRY_THREAD_STACKWALK_MODE_HUNG_ONLY has the same value as CRASHED_ONLY
and reads better for the app-hang option, where no thread crashed. The
app-hang option defaults to it.
@tustanivsky

Copy link
Copy Markdown
Collaborator Author

Would it make sense to reuse the existing sentry_thread_stackwalk_mode_t instead of a bool?

Yes, let's switch to that. I considered the enum at first too but held off for the first pass because of the confusion CRASHED_ONLY might cause with app hangs. The idea of adding an alias seems to address that concern 😅

tustanivsky added a commit to getsentry/sentry-unreal that referenced this pull request Nov 20, 2026
sentry-native replaced the bool option with
sentry_options_set_app_hang_thread_stackwalk_mode. Pin the sentry-native
submodule to getsentry/sentry-native#2327 until it is released.
@jpnurmi

jpnurmi commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

We could even try to come up with a better replacement for CRASHED_ONLY. "Single", "fault", "current", "target", ... something that fits both?

Also, what do you think about targeting v1? 🙈

@tustanivsky

Copy link
Copy Markdown
Collaborator Author

I was actually hoping to get this feature into Unreal sooner via a 0.x release. Would it be okay to follow up with another PR targeting v1 once this one lands to make the naming more generic?

@jpnurmi

jpnurmi commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Ok sure, no problem!

…reads

Log when the thread list cannot be built, instead of sending the event
without threads silently. Drop a Windows thread name that does not fit in
the buffer, since WideCharToMultiByte leaves a partial, possibly invalid
UTF-8 string. Read up to 4 KB of /proc/self/task/<tid>/status, so a long
Groups line cannot push SigPnd out of the buffer.
Comment thread src/sentry_app_hang_monitor.c Resolved
@tustanivsky
tustanivsky marked this pull request as ready for review November 20, 2026 15:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants