Python coprocess segfaults on the first dispatched request under CPython 3.12

Branch/Environment/Version

• Branch/Version: Release - reproduced on v5.8.6 and v5.13.2 (gateway images docker.tyk.io/tyk-gateway/tyk-gateway:v5.8.6 and :v5.13.2)
• Environment: On-prem, self-hosted gateway. Seen first on Kubernetes, then reduced to a local docker compose stack. No MDCB, no dashboard - use_db_app_configs: false, file-based API definitions.

Describe the bug

With a Python rich plugin ("driver": "python"), the gateway runs fine on CPython 3.10 and 3.11 but is unusable on 3.12. There are two separate problems behind each other.

1. The dispatcher bootstrap imports imp, which was removed in 3.12, so the gateway cannot start at all: /opt/tyk-gateway/coprocess/python/tyk/middleware.py calls imp.load_source() and /opt/tyk-gateway/coprocess/python/tyk/loader.py calls imp.load_module(..., imp.PY_SOURCE). These two files are byte-identical in v5.8.6, v5.13.2, v5.14.0 and a37d5e7f89232256eccc6efbdaf4c4283779d4d2.

2. Once those two calls are ported to importlib.util, the gateway boots, logs Python dispatcher was initialized and API Loaded, and then segfaults inside cgo on the first request that reaches the coprocess middleware. This is the substantive bug and the rest of this report is about it.

The cause of (2) is that dlpython calls the CPython C API from a thread that has no thread state, without ever acquiring the GIL. In dlpython/helpers.go, PyObjectGetAttr goes straight to the C API:

func PyObjectGetAttr(o unsafe.Pointer, attr interface{}) (unsafe.Pointer, error) {
	switch attr.(type) {
	case string:
		str := C.CString(attr.(string))
		defer C.free(unsafe.Pointer(str))
		pystr := PyUnicode_FromString(str)
		...

As far as I can see there is no PyGILState_Ensure in dlpython/helpers.go or in gateway/coprocess_python.go. Instead NewPythonDispatcher initialises the interpreter on a goroutine pinned with runtime.LockOSThread(), and DispatchWithContext serialises calls with a package-level Go mutex:

var (
	dispatcherClass    unsafe.Pointer
	dispatcherInstance unsafe.Pointer
	pythonLock         = sync.Mutex{}
)

A Go mutex does prevent two goroutines from entering the C API concurrently, but it does not give the calling OS thread a CPython thread state. The net/http goroutine that serves a request is on a different OS thread from the one that ran Py_Initialize, so tstate is NULL for that call.

This has always been undefined behaviour. It appears to have become fatal in 3.12 because object allocation now reaches per-interpreter state through the current thread state, so the NULL tstate is dereferenced at a small offset - which matches the faulting address we see, addr=0x10.

Reproduction steps

1. Build a gateway image with CPython 3.12 available and PYTHON_VERSION=3.12.
2. Port the two imp calls in /opt/tyk-gateway/coprocess/python/tyk/{middleware,loader}.py to importlib.util, otherwise the gateway dies at startup on problem (1) and never reaches this one. For middleware.py that means replacing imp.load_source(filepath, self.mw_path) with spec_from_file_location / module_from_spec / exec_module, and similarly for loader.py's imp.load_module.
3. Add an API with a Python plugin bundle - "driver": "python" and any hook. Ours uses an auth_check hook plus a response hook; the hook bodies should not matter, since the crash is in the dispatcher's attribute lookup before the plugin's own Python code runs.
4. Start the gateway. It comes up clean: Python version '3.12' loaded, Python dispatcher was initialized, API Loaded, no errors.
5. Send a single request to the API's listen path.

The gateway process dies. The client sees curl: (52) Empty reply from server and the container exits with code 2.

Minimal reproduction without Tyk

The same behaviour reproduces in a standalone C program with no Tyk involved, which is what convinced us the problem is the missing GIL acquisition rather than an ABI mismatch. Initialise the interpreter on the main thread, then call PyUnicode_FromString from a pthread with no PyGILState_Ensure, mirroring what dlpython does:

static void *worker(void *arg) {
    PyObject *s = PyUnicode_FromString("dispatch_hook");
    printf("worker: returned %p\n", (void *)s);
    return NULL;
}

int main(void) {
    Py_Initialize();
    pthread_t t;
    pthread_create(&t, NULL, worker, NULL);
    pthread_join(t, NULL);
    printf("SURVIVED\n");
    return 0;
}

The right-hand column also needs PyEval_SaveThread() after Py_Initialize(), otherwise PyGILState_Ensure in the worker deadlocks against the main thread, which still holds the GIL.

Actual behavior

The gateway boots successfully and then the whole process dies on the first request that is dispatched to the Python coprocess:

SIGSEGV: segmentation violation
PC=0x7feff0385d7e m=20 sigcode=1 addr=0x10
signal arrived during cgo execution

goroutine 56 gp=0xc000a5b880 m=20 mp=0xc000d80008 [syscall]:
runtime.cgocall(0x4671020, 0xc0021cc7b8)
	runtime/cgocall.go:167 +0x4b
github.com/TykTechnologies/tyk/dlpython._Cfunc_PyUnicode_FromString(0x7fefb0000ca0)
	_cgo_gotypes.go:495 +0x45
github.com/TykTechnologies/tyk/dlpython.PyUnicode_FromString(...)
	github.com/TykTechnologies/tyk/dlpython/binding.go:123
github.com/TykTechnologies/tyk/dlpython.PyObjectGetAttr(0x7fefd87e7aa0, ...)
	github.com/TykTechnologies/tyk/dlpython/helpers.go:126 +0xa5
github.com/TykTechnologies/tyk/gateway.(*PythonDispatcher).Dispatch(...)
	github.com/TykTechnologies/tyk/gateway/coprocess_python.go:51 +0xae
github.com/TykTechnologies/tyk/gateway.(*CoProcessor).Dispatch(0xc0021b84c0, 0xc000372e70)
	github.com/TykTechnologies/tyk/gateway/coprocess.go:659 +0x54
github.com/TykTechnologies/tyk/gateway.(*CoProcessMiddleware).ProcessRequest(...)
	github.com/TykTechnologies/tyk/gateway/coprocess.go:367 +0x76b
github.com/TykTechnologies/tyk/gateway.TraceMiddleware.ProcessRequest(...)
	github.com/TykTechnologies/tyk/gateway/middleware.go:105 +0x51a
...
net/http.(*conn).serve(0xc002092120, {0x54faad8, 0xc000c16240})
	net/http/server.go:2102 +0x625
created by net/http.(*Server).Serve in goroutine 117

On v5.13.2 the signature is identical - same addr=0x10, same PyObjectGetAttr -> PyUnicode_FromString frames; only the coprocess_python.go line number moves from 51 to 57.

In Kubernetes this presents as a crash-loop with no useful diagnostic, because the pod is restarted as soon as it takes its first request.

Expected behavior

A Python plugin should dispatch requests on CPython 3.12 as it does on 3.10 and 3.11, without terminating the gateway process. Failing that, the gateway should report an unsupported interpreter version at startup rather than accepting traffic and then dying on the first request.

As a temporary workaround, the docs should explicitly state that Python versions newer than 3.11 are currently not supported.

Screenshots/Video

N/A

Logs (debug mode or log file):

Problem (1), unmodified v5.8.6 on 3.12 - this is where the gateway stops before the patch:

{"level":"info","msg":"Loading API Specification from /mnt/tyk-gateway/apps/1.json"}
{"level":"info","msg":"----> Using bundle: bundle.zip","prefix":"main"}
{"level":"info","msg":"Python version '3.12' loaded","prefix":"coprocess"}
{"level":"error","msg":"No module named 'imp'","prefix":"python"}
{"level":"fatal","msg":"Couldn't initialize Python dispatcher","prefix":"coprocess"}

Problem (2), with the two imp calls ported to importlib.util. Startup is clean:

{"level":"info","msg":"Tyk API Gateway 5.13.2","prefix":"main"}
{"level":"info","msg":"Python version '3.12' loaded","prefix":"coprocess"}
{"level":"info","msg":"Python dispatcher was initialized","prefix":"coprocess"}
{"level":"info","msg":"API Loaded","api_id":"1","prefix":"gateway"}

then the first request produces the SIGSEGV traceback quoted above and the process exits with code 2. There is no log line between API Loaded and the fault.

Configuration (tyk config file):

Minimal config used for the local reproduction. The secrets here are dummy values from a throwaway test stack, not real credentials.

{
  "listen_port": 8080,
  "control_api_port": 9696,
  "secret": "sharedsecret",
  "node_secret": "sharedsecret",
  "template_path": "/opt/tyk-gateway/templates",
  "tyk_js_path": "/opt/tyk-gateway/js/tyk.js",
  "use_db_app_configs": false,
  "app_path": "/mnt/tyk-gateway/apps",
  "storage": {
    "type": "redis",
    "redis_host": "redis",
    "redis_port": 6379,
    "database": 0
  },
  "enable_analytics": false,
  "hash_keys": true,
  "hash_key_function": "sha256",
  "allow_insecure_configs": true,
  "coprocess_options": {
    "enable_coprocess": true,
    "python_path_prefix": "/opt/tyk-gateway"
  },
  "enable_bundle_downloader": true,
  "bundle_base_url": "http://middleware-server-test:8000"
}

Plugin manifest:

{
  "file_list": ["middleware.py"],
  "custom_middleware": {
    "auth_check": {"name": "AuthCheck"},
    "response": [{"name": "InjectCorrelationId"}],
    "driver": "python",
    "id_extractor": null
  },
  "checksum": "<injected at build time>",
  "signature": ""
}

Additional context

What we think the fix requires, in two parts:

1. Wrap the C-API entry points in dlpython in PyGILState_Ensure / PyGILState_Release, so a call from an arbitrary net/http thread has a valid thread state.
2. Release the GIL with PyEval_SaveThread() after Py_Initialize(), so those calls can actually acquire it. Without this, adding PyGILState_Ensure turns the crash into a deadlock against the initialising thread - we hit exactly that in the C reproduction above.

The Go-side pythonLock mutex would still be useful for serialisation, but it is not a substitute for holding the GIL.

Version matrix we measured, using a plugin hook that echoes a request header back so we can tell a real dispatch from a gateway-generated response:

Two notes that may save reviewer time:

• The importlib port is not the cause. It behaves identically to the unmodified dispatcher on 3.10, so holding Tyk and the dispatcher constant and changing only the interpreter from 3.10 to 3.12 is what introduces the crash.
• A boot-only test will look like a pass. The gateway reports Python dispatcher was initialized and API Loaded and stays up indefinitely; only an actual request reveals the fault.