sqlite3 converter lookup can crash during concurrent registry updates in free-threaded builds

[cuishuang]

Bug report

Bug description:

In Modules/_sqlite/cursor.c, _pysqlite_get_converter() currently looks up a converter with:

retval = PyDict_GetItemWithError(state->converters, upcase_key);
return retval;

PyDict_GetItemWithError() returns a borrowed reference.

In a free-threaded build, another thread can replace or remove the converter after the dictionary lookup has completed but before pysqlite_build_row_cast_map() appends the converter to row_cast_map.

The converter registry can be modified either through:

sqlite3.register_converter(name, replacement)

or directly through the public sqlite3.converters dictionary.

A possible sequence is:

1. A cursor thread obtains a borrowed converter reference.
2. Another thread replaces or removes that dictionary entry.
3. The previous converter is finalized.
4. The cursor thread passes the no-longer-valid reference to PyList_Append().

A stress test on a free-threaded debug build of CPython main reproduces an interpreter crash.

The adapter registry already uses PyDict_GetItemRef() when looking up adapters. The converter lookup should similarly retain a strong reference until the converter has been added to the row cast map.

A proposed fix is to make _pysqlite_get_converter() return a new reference using PyDict_GetItemRef() and update pysqlite_build_row_cast_map() to release that reference after PyList_Append().

Reproducer

import sqlite3
import threading

NAME = "CONVERTER_RACE"
NCOLUMNS = 64
NITER = 3000

QUERY = "select " + ", ".join(
    f"'value' as 'c{i} [{NAME}]'" for i in range(NCOLUMNS)
)

class Converter:
    __slots__ = ("value",)

    def __init__(self, value):
        self.value = value

    def __call__(self, value):
        return self.value

def reader():
    con = sqlite3.connect(
        ":memory:",
        detect_types=sqlite3.PARSE_COLNAMES,
    )
    try:
        for _ in range(NITER):
            con.execute(QUERY).fetchone()
    finally:
        con.close()

def mutator():
    for i in range(NITER * NCOLUMNS):
        sqlite3.register_converter(NAME, Converter(i))
        if i % 3 == 0:
            sqlite3.converters.pop(NAME, None)

threads = [
    *[threading.Thread(target=reader) for _ in range(8)],
    *[threading.Thread(target=mutator) for _ in range(2)],
]

for thread in threads:
    thread.start()
for thread in threads:
    thread.join()

CPython versions tested on:

CPython main branch

Operating systems tested on:

macOS

Linked PRs

• gh-155781: Hold strong references to sqlite3 converters #155784