loadDynlib() always uses RTLD_LOCAL, breaking dlopen() of extensions that need a package's own exported symbols (e.g. DuckDB)
Summary
src/js/dynload.ts's loadDynlib() -- used by both pyodide.loadPackage() and micropip.install() -- always calls Emscripten's loadDynamicLibrary() with local-only visibility (flags = 2, i.e. RTLD_NOW | RTLD_LOCAL). This means a loaded module's own exported symbols never reach Emscripten's shared/global symbol table, so a separate, later dlopen() call made from C/C++ code running inside that module (or any other already-loaded module) cannot resolve them -- even when the target module's WASM export section genuinely contains the symbol.
This is exactly the failure behind duckdb/duckdb-pyodide#2 ("extensions are not supported in the duckdb version of pyodide"), and I'd guess it affects any Pyodide package whose own compiled extension is designed to be a dlopen()-target for further, independently-loaded native code (plugin-style architectures), not just DuckDB.
Repro
- Install a real DuckDB Pyodide wheel (
duckdb.__version__confirmed working,import duckdbsucceeds). dlopen()(via DuckDB's own internal extension-loading machinery, triggered byLOAD <extension>;) a separately-built, ABI/version-matched.duckdb_extensionfile that needs symbols from_duckdb's own compiled module (e.g.duckdb::ExtensionLoader::GetDatabaseInstance()).- Fails:
Dynamic linking error: cannot resolve symbol _ZN6duckdb15ExtensionLoader19GetDatabaseInstanceEv.
Confirmed directly, not guessed:
- The symbol genuinely exists in
_duckdb's own compiled.so, checked two ways:llvm-nmshows it as a defined (T) symbol, andWebAssembly.Module.exports()on the raw bytes lists it as a real, exported function. pyodide._module.LDSO.loadedLibsByName["/lib/python.../site-packages/_duckdb..."].globalisfalseafter a normalimport duckdb(whether viapyodide.loadPackage(["duckdb"])ormicropip.install(<duckdb wheel url>)).
Root cause (in Emscripten, surfaced via Pyodide's own loader)
emscripten-core/emscripten's src/lib/libdylink.js, loadDynamicLibrary(): whether a load is global is carried per-DSO as dso.global = flags.global, and only global-flagged exports get merged into the shared symbol table (mergeLibSymbols) -- otherwise they land in a call-scoped localScope, invisible to any other, later dlopen() call. Pyodide's own dynload.ts hardcodes flags = 2 (i.e. global: false) for every package it loads via loadDynlib(), with no way for a package (or its installer) to opt into global visibility.
Workaround found, no Pyodide/Emscripten rebuild needed
loadDynamicLibrary() already has an "already loaded, promote to global" branch:
var dso = LDSO.loadedLibsByName[libName];
if (dso) {
if (!flags.global) { ... }
else if (!dso.global) {
dso.global = true;
mergeLibSymbols(dso.exports, libName) // promotes retroactively, no re-instantiation
}
...
}Calling pyodide._module.loadDynamicLibrary(key, {global: true, nodelete: true}) again, after the normal import duckdb, with the exact canonical key already present in LDSO.loadedLibsByName (to guarantee this branch is hit rather than a fresh, separate load), flips dso.global and merges the symbols in -- confirmed this resolves the dlopen() failure above completely, with a real end-to-end query then working.
This is reachable from pure Python too, not just JS glue on the host page -- pyodide_js (the Python-side FFI mirror of the JS pyodide controller object) exposes the same _module.loadDynamicLibrary/_module.LDSO, so a package's own Python code can self-heal this for its own dependencies without requiring anything from the embedding page.
Ask
Either of these would close the gap for anyone hitting this (not just DuckDB):
- A documented, supported way to request
RTLD_GLOBAL-equivalent loading for a specific package viapyodide.loadPackage()/micropip.install()(an option, rather than the hardcodedflags = 2). - At minimum, documentation of the workaround above (the
pyodide_js._module.loadDynamicLibrary(key, {global: true})re-promotion pattern) somewhere discoverable, since as far as I can tell this isn't written down anywhere today and duckdb-pyodide#2's own conclusion (linked above) reads as "not supported" rather than "here's how."
Happy to open a PR for either the doc or a global passthrough option if that's welcome -- wanted to raise it here first since I don't know this codebase's conventions for exposing new loader flags.
Source: pyodide/pyodide