`tracemalloc` raw-domain allocator hook violates `PYMEM_DOMAIN_RAW` contract

[chirizxc]

Related:

• GlobalAlloc that uses PyMem_Raw* methods PyO3/pyo3#6268

• GlobalAlloc that uses PyMem_Raw* methods PyO3/pyo3#6268 (comment)

• add PyMemRawAllocator PyO3/pyo3#6279

Bug description:

tracemalloc's raw-domain allocator hook requires an attached thread
state (PyGILState_Ensure(): tracemalloc_alloc / tracemalloc_realloc) , which violates the documented PYMEM_DOMAIN_RAW contract:

• https://docs.python.org/3.15/c-api/memory.html#allocator-domains

• https://docs.python.org/3.15/c-api/memory.html#raw-memory-interface

• https://docs.python.org/3.15/c-api/memory.html#raw-memory-interface:~:text=PyMem%5FSetAllocator%28%29%20does%20have%20the%20following%20contract

MRE (Minimal Reproducible Example):

Pure Python MRE

import ctypes
import sys
import threading
import tracemalloc

if not sys.platform.startswith("linux"):
    msg = "This MRE requires Linux"
    raise RuntimeError(msg)

pythonapi = ctypes.pythonapi
pythonapi.PyMem_RawMalloc.restype = ctypes.c_void_p
pythonapi.PyMem_RawMalloc.argtypes = [ctypes.c_size_t]
pythonapi.PyMem_RawFree.argtypes = [ctypes.c_void_p]


# this is only needed to creat a native thread
# without a `PyThreadState` attached to it
libc_cdll = ctypes.CDLL(None)
libc_cdll.pthread_create.restype = ctypes.c_int
libc_cdll.usleep.argtypes = [ctypes.c_uint]

# unlike CDLL, PyDLL does NOT release the GIL around the call
# this is intentional, so that the main thread holds the GIL for
# the full 5 seconds creating conditions under which the raw-allocator
# hook cannot acquire the GIL, even if it were to request it
blocking_libc = ctypes.PyDLL(None)
blocking_libc.sleep.argtypes = [ctypes.c_uint]

THREAD_FN = ctypes.CFUNCTYPE(ctypes.c_void_p, ctypes.c_void_p)
done = threading.Event()


@THREAD_FN
def native_thread_entry(_arg: ctypes.c_void_p) -> ctypes.c_void_p:
    # this thread was created using `pthread_create()`, not
    # `threading.Thread` it does NOT have an associated `PyThreadState`
    libc_cdll.usleep(500_000)

    # according to the `PyMem_SetAllocator` contract for `PYMEM_DOMAIN_RAW`
    # ("a thread state is not attached when the allocator is called"),
    # the call below MUST execute without blocking on the GIL
    ptr = pythonapi.PyMem_RawMalloc(64)
    pythonapi.PyMem_RawFree(ptr)

    # if the allocator internally calls `PyGILState_Ensure()`,
    # then this code will never reach this point while the GIL is busy
    done.set()


def main() -> None:
    # tracemalloc.start() sets a hook on `PYMEM_DOMAIN_RAW`
    tracemalloc.start()
    thread_id = ctypes.c_ulong()
    libc_cdll.pthread_create(ctypes.byref(thread_id), None, native_thread_entry, None)
    # The main thread holds the GIL continuously for 5 seconds (`PyDLL.sleep` does not
    # release the GIL). If the raw-allocator hook requires the GIL, a deadlock occurs
    blocking_libc.sleep(5)

    # deadlock
    print("DEADLOCK" if not done.is_set() else "OK")


main()

mre.c:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

#include <pthread.h>
#include <stdatomic.h>
#include <stdbool.h>
#include <unistd.h>

static atomic_bool done = false;

static void *native_thread_entry(void *arg)
{
    (void)arg;

    usleep(500000);

    void *ptr = PyMem_RawMalloc(64);
    PyMem_RawFree(ptr);

    atomic_store(&done, true);
    return NULL;
}

static PyObject *mre_run(PyObject *self, PyObject *args)
{
    (void)self;
    (void)args;

    atomic_store(&done, false);

    pthread_t thread;
    int err = pthread_create(&thread, NULL, native_thread_entry, NULL);
    if (err != 0) {
        return PyErr_SetFromErrno(PyExc_OSError);
    }

    sleep(5);

    printf("%s\n", atomic_load(&done) ? "OK" : "DEADLOCK");

    pthread_detach(thread);

    Py_RETURN_NONE;
}

static PyMethodDef methods[] = {
    {
        "run",
        mre_run,
        METH_NOARGS,
        "Run the raw allocator MRE.",
    },
    {NULL, NULL, 0, NULL},
};

static struct PyModuleDef module = {
    PyModuleDef_HEAD_INIT,
    "mre",
    NULL,
    -1,
    methods,
};

PyMODINIT_FUNC PyInit_mre(void)
{
    return PyModule_Create(&module);
}

setup.py:

from setuptools import Extension, setup

setup(
    name="mre",
    ext_modules=[
        Extension(
            "mre",
            sources=["mre.c"],
            extra_compile_args=["-std=c11"],
        ),
    ],
)

test.py:

import tracemalloc

import mre

tracemalloc.start()
mre.run()

CPython versions tested on:

3.10, 3.11, 3.12, 3.13, 3.14, 3.15

Operating systems tested on:

Linux

[ZeroIntensity]

The definition of tracemalloc's raw allocator is here:

cpython/Python/tracemalloc.c

Lines 688 to 692 in 37e98da

	static void*
	tracemalloc_raw_malloc(void *ctx, size_t size)
	{
	return tracemalloc_alloc(1, 0, ctx, 1, size);
	}

In tracemalloc_alloc, this sets need_gil to 1, which will in turn call PyGILState_Ensure to get an attached thread state:

cpython/Python/tracemalloc.c

Lines 499 to 501 in 37e98da

	static void*
	tracemalloc_alloc(int need_gil, int use_calloc,
	void *ctx, size_t nelem, size_t elsize)

So, the problem isn't that tracemalloc's raw allocator needs the caller to hold the GIL (if it did, I suspect you'd see a crash at runtime when calling PyMem_RawMalloc). What's the expected behavior of the MRE? I see it prints "DEADLOCK", but the program finishes.

[chirizxc]

What's the expected behavior of the MRE?

DEADLOCK

[chirizxc]

The PYMEM_DOMAIN_RAW contract is still violated: the documentation explicitly requires that "a thread state is not attached when the allocator is called", and calling PyGILState_Ensure() to obtain that state is a hidden, undocumented block on the GIL inside a call that is supposed to be safe without it. In the MRE, this manifests not as a "permanent deadlock" but as:

• An unpredictable delay in PyMem_RawMalloc() / PyMem_RawFree(), lasting up to however long another thread holds the GIL (up to 5 seconds in the example), which by itself violates the contract that the allocator "can operate without an attached thread state," i.e., without blocking on the GIL at all.

• A potential genuine deadlock in more complex scenarios, for example, if the thread holding the GIL is itself waiting on the result of an operation that depends on PyMem_RawMalloc() completing in the native thread (a classic lock-ordering inversion deadlock), or if the GIL is held indefinitely (not just 5 seconds, but say, until a long blocking operation in a C extension finishes).

[ZeroIntensity]

The contract doesn't specify that a thread state can't be attached inside the callback. It only specifies that the callback isn't called with one.

A potential genuine deadlock in more complex scenarios, for example, if the thread holding the GIL is itself waiting on the result of an operation that depends on PyMem_RawMalloc() completing in the native thread (a classic lock-ordering inversion deadlock), or if the GIL is held indefinitely (not just 5 seconds, but say, until a long blocking operation in a C extension finishes).

A thread shouldn't be holding the GIL at all if it's waiting on some operation (except in some very special circumstances).

[chirizxc]

It seems more accurate to say not that the contract is "violated," but that the implementation makes optional a blocking call for something the documentation explicitly names as the reason the raw domain exists in the first place - an allocator that "can operate without an attached thread state". In other words, the hook does technically work (it doesn't crash, and there's no UB), but it stops being non-blocking / GIL-independent in cases where the GIL is busy - this is a regression in expected behavior/performance, not undefined behavior or a literal violation of the letter of the spec.

[ZeroIntensity]

Yes, but looking at the code, changing tracemalloc to not need the GIL would likely be a significant change. I guess we could try swapping out PyGILState_Ensure calls for PyMutex calls to a static lock, but I don't know what implications that would have, or if that's even possible (if tracemalloc is calling into something that needs the GIL, such as incrementing an object's reference count, then there's definitely no easy way to fix this).

[emmatyping]

changing tracemalloc to not need the GIL would likely be a significant change

FWIW I expect tracemalloc will need to be re-architected for free-threaded and subinterpreters. There are thread-safety issues (e.g. #126315) and issues with sub-interpreters (e.g. #134604). I expect to fix both tracemalloc will need a fair bit of work to be functional in those scenarios.

[ZeroIntensity]

Honestly, I think it would be better to deprecate and remove tracemalloc. It's slow, buggy, and not very useful. Memray is better in basically every way.

[emmatyping]

Yeah. Even with a re-architecture, unless we are okay with a lot more <unknown> frames, an attached thread state is needed to get the current thread's Python stack frames. I suppose we could queue acquiring the stack trace until the next time the interpreter re-attaches? But I'd need to think more if that can be done safely.

[emmatyping]

I do think an allocation in another thread causing a deadlock with tracemalloc is pretty bad. We should at very minimum document that this may happen and ideally conservatively collect Python frame information for PYMEM_DOMAIN_RAW (either not require a threadstate or defer collecting the trace).

[ZeroIntensity]

I'm still not sure I understand the scenario where this will deadlock. Why is a thread waiting on an allocation without releasing the GIL?

[ngoldbaum]

PyMem_RawMalloc isn't documented as a possibly-blocking API. It's also not blocking in-practice except when tracemalloc is enabled. I would only think about discipline around releasing the GIL if I'm calling a possibly-blocking API.

Usually when an API is documented as being usable without an attached thread state you can also use it with a thread state. In this case that's not true if tracemalloc is enabled.