Audit hooks added before Py_Initialize() are freed with the wrong allocator, crashing at exit #161407

Crash report

I've been looking into the performance of streaming decompression with compression.zstd, first on 3.14 and 3.15, then on the free-threaded builds. That turned up a memory issue in free-threaded builds, which I'm reporting separately. This crash is something I ran into along the way.

While reproducing the memory issue on a Windows machine, the runs with PYTHONMALLOC=mimalloc on 3.15 crashed at exit with an access violation, after the script itself had finished fine. Even python -c pass crashed. 3.14 didn't, and neither did any run without PYTHONMALLOC.

It turned out the machine's endpoint security agent loads into python.exe and registers an audit hook with PySys_AddAuditHook() before Py_Initialize(). That's allowed: the docs say the function "is safe to call before Py_Initialize". But the hook's list entry is allocated with PyMem_RawMalloc() at that point, before PYTHONMALLOC is applied, so it comes from malloc(). At finalization, _PySys_ClearAuditHooks() frees it with PyMem_RawFree(), which by then is mimalloc's. mi_free() on a block from the C heap crashes.

Anything that changes the raw allocator at startup has the same problem:

• PYTHONMALLOC=mimalloc on builds with the GIL, since 3.15 (before, it left the raw domain on malloc());

• the debug hooks, since 3.13 at least: -X dev / PYTHONDEVMODE=1 and the *_debug allocators. There the crash is sometimes caught as

  Fatal Python error: _PyMem_DebugRawFree: bad ID: Allocated using API '', verified using API 'r'
  

I think #161074 is this bug: a crash in audit-hook cleanup under -X dev on Windows, on a machine with a security product installed, closed without a root cause.

Reproducer

You don't need an agent for this. CPython's own embedding test program adds a hook before Py_Initialize(), and in a source build it crashes the same way:

$ PYTHONMALLOC=mimalloc ./Programs/_testembed test_audit; echo $?
138
$ PYTHONDEVMODE=1 ./Programs/_testembed test_audit; echo $?
139

(main with the GIL on macOS; a bus error and a segmentation fault.) test_embed only runs this scenario with the default allocator, which is probably why CI hasn't caught it.

Against an installed Python, a few lines do the same:

#include <Python.h>

static int
hook(const char *event, PyObject *args, void *data)
{
    return 0;
}

int
main(void)
{
    PySys_AddAuditHook(hook, NULL);
    Py_Initialize();
    PyRun_SimpleString("import sys; print(sys.version.split()[0], 'ran')");
    return Py_FinalizeEx() < 0 ? 120 : 0;
}

$ cc -o embed embed.c -I$PREFIX/include/python3.15 -L$PREFIX/lib -lpython3.15 -Wl,-rpath,$PREFIX/lib
$ PYTHONHOME=$PREFIX PYTHONMALLOC=mimalloc ./embed; echo $?
3.15.2 ran
138

($PREFIX is a python-build-standalone 3.15.2 on macOS.)

Why 3.15

In 3.14, set_up_allocators_unlocked() in Objects/obmalloc.c kept malloc() for the raw domain under PYTHONMALLOC=mimalloc:

    case PYMEM_ALLOCATOR_MIMALLOC:
    case PYMEM_ALLOCATOR_MIMALLOC_DEBUG:
    {
        PyMemAllocatorEx malloc_alloc = MALLOC_ALLOC;
        set_allocator_unlocked(PYMEM_DOMAIN_RAW, &malloc_alloc);

Since 3.15 it switches the raw domain to mimalloc as well:

        PyMemAllocatorEx malloc_alloc = MIMALLOC_RAWALLOC;
        set_allocator_unlocked(PYMEM_DOMAIN_RAW, &malloc_alloc);

Free-threaded builds use mimalloc for the raw domain from the start, so for them only the debug hooks trigger it.

What I tested

macOS 26 (arm64) and Windows 11 (x64), python-build-standalone builds and main (aaaf61c):

I ran -X dev on macOS only, and haven't tried Linux. Without PYTHONMALLOC or dev mode, nothing crashes.

Proposed fix

Allocate and free the entries with plain malloc() / free() (_PyMem_RawMalloc(NULL, ...) and _PyMem_RawFree(NULL, ...)), which nothing at startup replaces, and add a test_embed test that runs _testembed test_audit under the affected settings.

I first tried _PyMem_DefaultRawMalloc() / _PyMem_DefaultRawFree(), as other pre-initialization data does. That fixes release builds, but a Py_DEBUG build still crashed under PYTHONMALLOC=mimalloc_debug, because there they go through the debug hooks, whose underlying allocator gets replaced.

With the patch below, _testembed test_audit exits cleanly under every PYTHONMALLOC value and in dev mode: on macOS in a release build with the GIL, a free-threaded build and a Py_DEBUG build, and on Windows in release builds with and without the GIL. (Windows ran the first version, which is the same code in release builds.) The new test fails without the patch and passes with it; test_audit, test_embed, test_sys and test_capi.test_mem pass. I'm happy to open a PR.

diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py
index 773b97f..caeebec 100644
--- a/Lib/test/test_embed.py
+++ b/Lib/test/test_embed.py
@@ -2062,6 +2062,24 @@ def test_open_code_hook(self):
     def test_audit(self):
         self.run_embedded_interpreter("test_audit")
 
+    def test_audit_hook_survives_allocator_change(self):
+        # A hook added before Py_Initialize() is freed at finalization, after
+        # PYTHONMALLOC changed the raw allocator: the entry must not be
+        # allocated with one allocator and freed with another.
+        allocators = ['debug']
+        if not support.Py_GIL_DISABLED:
+            allocators.append('malloc_debug')
+        if support.with_mimalloc():
+            allocators.extend(['mimalloc', 'mimalloc_debug'])
+        for allocator in allocators:
+            with self.subTest(allocator=allocator):
+                self.run_embedded_interpreter(
+                    "test_audit", env={'PYTHONMALLOC': allocator})
+        # The development mode installs the debug hooks too
+        with self.subTest(dev_mode=True):
+            self.run_embedded_interpreter(
+                "test_audit", env={'PYTHONDEVMODE': '1'})
+
     def test_audit_tuple(self):
         self.run_embedded_interpreter("test_audit_tuple")
 
diff --git a/Python/sysmodule.c b/Python/sysmodule.c
index 6f336da..036c92e 100644
--- a/Python/sysmodule.c
+++ b/Python/sysmodule.c
@@ -32,6 +32,7 @@ Data members:
 #include "pycore_pylifecycle.h"   // _PyErr_WriteUnraisableDefaultHook()
 #include "pycore_pymath.h"        // _PY_SHORT_FLOAT_REPR
 #include "pycore_pymem.h"         // _PyMem_DefaultRawFree()
+#include "pycore_pymem_init.h"    // _PyMem_RawMalloc()
 #include "pycore_pystate.h"       // _PyThreadState_GET()
 #include "pycore_pystats.h"       // _Py_PrintSpecializationStats()
 #include "pycore_runtime.h"       // _PyRuntimeState_Get*()
@@ -448,7 +449,7 @@ _PySys_ClearAuditHooks(PyThreadState *ts)
     runtime->audit_hooks.head = NULL;
     while (e) {
         n = e->next;
-        PyMem_RawFree(e);
+        _PyMem_RawFree(NULL, e);
         e = n;
     }
 }
@@ -496,8 +497,12 @@ PySys_AddAuditHook(Py_AuditHookFunction hook, void *userData)
         }
     }
 
-    _Py_AuditHookEntry *e = (_Py_AuditHookEntry*)PyMem_RawMalloc(
-            sizeof(_Py_AuditHookEntry));
+    /* Use malloc(), not PyMem_RawMalloc(): a hook can be added before
+       Py_Initialize(), and the entry is freed during finalization, after
+       PYTHONMALLOC may have changed the raw allocator (and, in debug builds,
+       the allocator under the debug hooks). */
+    _Py_AuditHookEntry *e = (_Py_AuditHookEntry*)_PyMem_RawMalloc(
+            NULL, sizeof(_Py_AuditHookEntry));
     if (!e) {
         if (tstate != NULL) {
             _PyErr_NoMemory(tstate);

CPython versions tested on:

3.13, 3.14, 3.15, CPython main branch

Operating systems tested on:

macOS, Windows

Output from running 'python -VV' on the command line:

Python 3.15.2 (main, Oct 26 2026, 13:59:33) [Clang 23.1.3 ]

Linked PRs

• gh-161407: Allocate audit hook entries with malloc() #161409