Decouple test suite from compiler internals for pure blackbox testing
Author: sbc100Created Sep 4, 2026Updated Sep 4, 2026
Labelstests
Most tests in test/test_core.py and test/test_browser.py test emcc as a black box via CLI invocations. However, parts of test/test_other.py, test/test_sanity.py, and the test harness (test/common.py) directly import and depend on Python compiler internals (tools/*).
If we want to rewrite the compiler (or components like the linker) in another language, or run the test suite against alternative implementations, we need to eliminate these internal dependencies and make the suite purely blackbox.
Breakdown & Estimated Effort
| Area | Current Coupling | Proposed Fix | Effort |
|---|---|---|---|
| 1. Global State Mutation | Tests for -print-* flags in test_other.py mutate options.lto and settings.MEMORY64 to predict paths via cache.get_lib_dir(). |
Assert on CLI output paths directly without altering Python singletons. | Small (~0.5 day) |
| 2. Binary Type Checks | test_other.py calls building.is_wasm(), is_wasm_dylib(), is_ar(). |
Inspect magic bytes (b'\0asm') or use llvm-readobj / llvm-ar. |
Small (~1 day) |
| 3. Direct Unit Tests | test_other.py / test_browser.py unit-test Python functions directly (binary_encode, wasm_sourcemap, response_file, closure_compiler). |
Test via CLI flags or isolate into a dedicated Python unit test directory. | Small (~1–2 days) |
| 4. Wasm Inspection | ~15 tests in test_core.py and test_other.py use tools.webassembly.Module to check section names/features. |
Replace with standard CLI tools (llvm-readobj, llvm-objdump -h, wasm-dis). |
Medium (~2–3 days) |
| 5. Cache & Ports Introspection | Tests query cache.get_sysroot_dir() or ports.Ports.get_dir() to find files in the cache. |
Expose paths via CLI (e.g. emcc --print-sysroot) and test ports via --use-port. |
Medium (~2 days) |
| 6. Harness Settings Filtering | common.py imports COMPILE_TIME_SETTINGS to filter -s flags during compile-only (-c) runs. |
Make compiler ignore/warn on link settings during -c, or query capabilities via CLI. |
Medium (~2–3 days) |
| 7. Sanity Suite Partitioning | test_sanity.py (~1,000 lines) tests the Python toolchain driver (config.py, cache locking, version checks). |
Separate Python driver tests from the language-agnostic conformance test suite. | Medium (~1–2 days) |
Total Estimated Effort: ~2 to 3 weeks
Next Steps
- Clean up quick wins in
test_other.py(removeoptions/settingsmutation, replacebuilding.is_*helpers). - Replace
tools.webassemblycalls with standard binary tools (llvm-readobj). - Decouple harness dependencies (
COMPILE_TIME_SETTINGS) and separatetest_sanity.py.
Source: emscripten-core/emscripten