Skip to content

Fix DynamicLibrary test hang on the VS 2026 CI runner - #230

Merged
BenjaFriend merged 1 commit into
mainfrom
fix/msvc2026-dynlib-hang
Oct 29, 2026
Merged

BenjaFriend merged 1 commit into
mainfrom
fix/msvc2026-dynlib-hang

Conversation

@BenjaFriend

Copy link
Copy Markdown
Member

Fixes the red main CI since #220: both Windows MSVC 2026 jobs time out on the DynamicLibrary test.

Root cause

The test loads a fake 4-byte "DLL" through DynamicLibrary::Open. LoadLibraryW on a bad image opens a modal "Bad Image" (0xc0000130) dialog unless the error mode has SEM_FAILCRITICALERRORS. The process error mode comes from the parent:

  • Git Bash and CTest 3.x set it, so the dialog never showed (VS 2022 runner, local runs, and the direct runs in Diag/msvc2026 dynlib #229).
  • CTest 4.x (4.4.4 on the windows-2025-vs2026 image) does not, so the dialog opens on a headless runner and the test waits until the 300 s timeout.

Reproduced locally with the same CoreTests.exe: it passes directly and under CTest 3.29, and shows the dialog and hangs under VS 2026's CTest 4.3.1.

Fix

  • Open and OpenSigned set SEM_FAILCRITICALERRORS with SetThreadErrorMode around the load and restore the caller's mode after. This is per thread, so no process-wide state changes. A corrupt DLL in a shipped game no longer blocks on a dialog either.
  • New test section "A bad image does not open a dialog": it sets the thread error mode to 0 (dialogs on), loads a bad image, and checks that the mode is restored. The test no longer depends on the launcher to hide the bug.

Verified locally (VS 2026, Debug)

  • DynamicLibrary passes under CTest 4.3.1 (0.03 s), under CTest 3.29, and run directly.
  • clang-format and check_comment_style.py are clean.

#229 (the diagnostic branch) can be closed after this merges.

🤖 Generated with Claude Code

LoadLibrary on a file that is not a valid DLL opens a modal "Bad Image"
dialog unless the error mode has SEM_FAILCRITICALERRORS. The process error
mode comes from the parent. Git Bash and CTest 3 set it, CTest 4 does not,
so on the VS 2026 runner (CTest 4.4) the DynamicLibrary test waited for a
click until the CTest timeout.

Open and OpenSigned now set the error mode of the calling thread around
the load and restore it after. A new test turns dialogs on for its thread
before it loads a bad image, so the test does not depend on the launcher.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BenjaFriend
BenjaFriend merged commit 12d25f5 into main Oct 29, 2026
12 checks passed
@BenjaFriend
BenjaFriend deleted the fix/msvc2026-dynlib-hang branch October 29, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant