Fix intermittent teardown crash in the cDAC GC stress framework#130526
Open
max-charlamb wants to merge 2 commits into
Open
Fix intermittent teardown crash in the cDAC GC stress framework#130526max-charlamb wants to merge 2 commits into
max-charlamb wants to merge 2 commits into
Conversation
max-charlamb
pushed a commit
to max-charlamb/runtime
that referenced
this pull request
Jul 10, 2026
The shutdown/teardown crash fix in src/coreclr/vm/cdacstress.cpp is being tracked separately in dotnet#130526 so it can land independently of the cDAC arg-GCRefMap changes in this PR. This restores cdacstress.cpp to its upstream state here. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc4ebcc7-53f1-47e9-a84e-830b48bd07b2
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adjusts the CoreCLR cDAC GC stress shutdown path to avoid teardown-time crashes by disabling the stress trigger gate earlier, draining in-flight verifications under s_cdacLock, and intentionally leaking the cDAC reader handle/module at process exit rather than freeing/unloading them during EE shutdown.
Changes:
- Disable stress triggers (
s_initialized/s_cdacStressLevel) before teardown, then acquires_cdacLockto ensure any in-flightVerifyAtStressPointcompletes before releasing shared state. - Preserve accurate summary reporting by snapshotting which sub-checks were enabled before the shutdown gate is disabled.
- Stop calling
cdac_reader_freeduring shutdown (and avoid unloading the cDAC module), to prevent teardown races with background stress-hook threads.
Contributor
|
Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag |
0f8ae6f to
4d3adf9
Compare
4d3adf9 to
958543d
Compare
958543d to
9d1ffd5
Compare
9d1ffd5 to
961fd13
Compare
The cDAC stress harness crashed intermittently at process teardown on arm/arm64: each run completed all verifications and printed its summary (e.g. MultiThread ARGITER "84 pass / 0 fail"), then the process exited 139 (SIGSEGV). The uploaded per-debuggee logs show the crash is after verification finishes, during shutdown. CdacStressPolicy::Shutdown runs from EEShutDownHelper (g_fEEShutDown = ShutDown_Start is set immediately before it, and the finalizer/thread rendezvous happens after), so background threads may still be allocating -- the MultiThread debuggee spins up worker threads. Such a thread can pass the IsEnabled() gate and enter VerifyAtStressPoint, which uses the cDAC reader and cached COM interfaces, exactly as Shutdown releases/frees them. Fix: Shutdown holds s_cdacLock across the entire teardown and clears s_initialized under it (the reader is still freed via cdac_reader_free as before). This drains any in-flight VerifyAtStressPoint and blocks new ones, which re-check s_initialized under the same lock and bail, so none runs against released state. s_cdacLock's acquire/release ordering publishes the transition, so no volatile accesses are needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc4ebcc7-53f1-47e9-a84e-830b48bd07b2
961fd13 to
a554e1a
Compare
rcj1
approved these changes
Jul 10, 2026
Member
Author
|
/azp run runtime-diagnostics Note This comment was generated with the assistance of GitHub Copilot. |
|
No pipelines are associated with this pull request. |
src/coreclr/vm/cdacstress.cpp is the native cDAC GC stress harness that the runtime-diagnostics CdacStressTest legs exercise, but it was not in the pipeline's pr path filter, so changes to it (like the shutdown fix in this PR) did not trigger the very legs that validate them. Add it to the include list. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: dc4ebcc7-53f1-47e9-a84e-830b48bd07b2
steveisok
approved these changes
Jul 11, 2026
This was referenced Jul 11, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes an intermittent crash at process teardown in the cDAC GC stress framework, seen on arm/arm64 as
exit 139(SIGSEGV). Each affected run completes all verifications and prints its summary (e.g. MultiThread ARGITER84 pass / 0 fail), then crashes during shutdown -- so the fault is in teardown, not in the verification logic.Split out of #130447 so it can land independently.
Root cause
CdacStressPolicy::Shutdownruns fromEEShutDownHelperbefore the finalizer/thread rendezvous, so background threads can still be allocating (the MultiThread debuggee spins up worker threads). Such a thread can pass theIsEnabled()gate and enterVerifyAtStressPoint, which uses the cDAC reader and cached COM interfaces, exactly asShutdownreleases/frees them -- a use-after-release.Fix
Shutdownholdss_cdacLockacross the whole teardown and clearss_initializedunder it.VerifyAtStressPointre-checkss_initializedunder the same lock and bails, so no verification runs against state being torn down.cdac_reader_freeis still called as before.Testing
The crash is arm/arm64-only (does not reproduce on x64), so only CI can validate it. An x64 MultiThread stress soak (80 runs) passes with summaries intact and this change does not alter the x64 behavior.
Note
This PR description and change were produced with the assistance of GitHub Copilot.