Faster config loading: an antlr4-free alternative to `OmegaConfigLoader`

Introduction

Config loading is a measured, user-reported bottleneck for large Kedro projects, and a blocker for thread-safe config sharing. Both constraints sit inside OmegaConf rather than inside Kedro. OmegaConfigLoader (OCL) uses a narrow slice of OmegaConf — parse YAML/JSON, deep-merge files, resolve ${...}, return plain dicts — but pays for the full object model and the antlr4 parser runtime, which parses every ${...} occurrence at resolution time.

I built deepyaml: a pure-Python implementation of exactly that slice, with no antlr4 and PyYAML as its only dependency. It ships a DeepYAMLConfigLoader that drops into CONFIG_LOADER_CLASS, and it is 11–31× faster end-to-end on generated Kedro-shaped config trees while producing type-identical output to OCL across a scenario-by-scenario equivalence suite. All numbers and tests below were re-run against kedro 1.6.0, released 2026-09-09.

What I'm asking for in this session: feedback on the approach, and agreement on how to validate it. I am not proposing this become part of Kedro core today. Specifically:

1. Is the performance claim measured the right way? (See Testing — I'd like to settle on Kedro's own asv suite as the yardstick.)
2. Is the fidelity strategy — an executable equivalence suite plus a published divergence list — a credible substitute for "it is OmegaConf"?
3. What would have to be true for you to be comfortable pointing a real project at this?

Background

The performance problem is documented and still open in substance

• #3893 — users reporting ~3-minute config loads, and kedro.config accounting for >50% of pipeline run duration.
• #4322 — narrowed the cost to interpolation-heavy catalogs, naming OmegaConf.load, OmegaConf.to_container, and OmegaConf.merge, with to_container (i.e. resolution) the dominant term.
• #4367 — took the wins available inside OmegaConf (stop re-creating _globals/runtime_params per reference; separate the merge from the underscore filter) and states plainly that the bottleneck remains those three OmegaConf calls.

That last point is the premise of this proposal: the residual cost is structural. antlr4 parses each interpolation expression at resolution time, so the cost scales with interpolation density and cannot be optimized away from the Kedro side.

A second, independent problem: thread safety (#5660)

OmegaConfigLoader.__getitem__ calls OmegaConf.register_new_resolver("runtime_params", ..., replace=True) on every access, and that registry is process-global. Concurrent loaders with different runtime_params therefore overwrite one another — the blocker for sharing pre-loaded configuration in serving use cases (#5535, #5660).

Measured here: 200 concurrent loads, each from its own loader with distinct runtime_params.

The failure mode is slightly worse than #5660 describes. As well as silently returning another thread's value, a thread can pick up another's globals-mode resolver and raise UnsupportedInterpolationType on a perfectly valid config.

deepyaml has no global registry to clobber: _resolver_map() builds a fresh mapping of instance-bound callables per get() call. A single loader shared across 8 threads also came back clean over 2,000 interleaved loads — though I'd scope that to what I've measured rather than claim thread safety in general.

This matters for the proposal because contextvars, the fix suggested on #5660, is a workaround for OmegaConf's global registry rather than something Kedro can design away. Same shape as the performance argument: the constraint lives in the dependency.

Approaches I tried previously

1. Rust (PyO3 + maturin) on top of config-rs or figment — rejected. Both crates are built for "merge layered sources → deserialize into a fixed typed struct." Neither has an interpolation engine, which is the actual hard part, and neither's merge semantics match Kedro's. The substrate would have saved the easy half and left the hard half.
2. A measurement spike before committing to a language. Profiling OmegaConf doing OCL's work: resolution ≈60% of total cost, merge the smallest term. So removing antlr4 is essentially the entire win, and a pure-Python implementation captures it. Rust's only real remaining edge is parsing, which is already libyaml-bound and rentable as a dependency. This is why deepyaml is pure Python — the Rust version would have added platform wheels and a WASM tax for a fraction of the gain.
3. Scoping down from "replace OmegaConf" to "replace what OCL needs." OCL never exposes a DictConfig — it does load → merge → resolve → to_container → plain dict. That makes the replacement surface small enough to reimplement faithfully and test exhaustively: no structured configs, no object model, no Hydra ambitions.

Secondary motivation: WASM/Pyodide

OmegaConf 2.3.1 — still the current stable release, and what Kedro installs today — pins antlr4-python3-runtime==4.9.*, and 4.9.3 is published as an sdist only, so micropip cannot build it and OmegaConf does not install under Pyodide. That blocks Kedro in browser environments today.

This particular argument has a shelf life, and I'd rather say so than have it discovered: the unreleased 2.4 line (2.4.0.dev15, uploaded 2026-08-07) vendors antlr4 inside the package rather than depending on it, so there is no sdist-only dependency left to build and the installability problem goes away. Vendoring does not make it faster — see What about OmegaConf 2.4? below, where I measure it.

deepyaml depends only on PyYAML, which falls back to the pure-Python loader where libyaml is unavailable. I mention WASM as a bonus, not as the argument.

Problem

Loading configuration is disproportionately expensive for projects with large catalogs and heavy ${...} use, and the cost is concentrated in a dependency Kedro uses for a fraction of its capability.

What's in scope

• The OmegaConfigLoader pipeline: pattern-based file discovery, per-environment parse and deep-merge, ${...} resolution, and environment layering, returning plain dicts.
• Local filesystem config sources, YAML and JSON.
• Bug-for-bug fidelity with OCL's observable behavior, including its YAML dialect quirks.

What's not in scope

• Being a general OmegaConf drop-in: no DictConfig object model, no structured/dataclass configs, no Hydra-style composition.
• Remote (s3://, http://) and archive (.tar, .zip) conf sources — see Future iterations.
• Any change to Kedro core, or to OCL's public API, at this stage. Adoption today is entirely opt-in via CONFIG_LOADER_CLASS.
• Changing configuration semantics. Anywhere deepyaml and OCL disagree, that is a bug in deepyaml unless it appears in the divergence list below.

Design

Pipeline

Four stages, each an independently testable pure function, with the loader supplying policy:

parse (libyaml)  →  deep-merge  →  resolve ${...}  →  plain dict

617 lines of code (947 including docstrings) across 8 modules; PyYAML is the only runtime dependency. The Kedro adapter is 68 lines of it.

The interpolation engine

The substantive claim is that OmegaConf's grammar as Kedro configs actually use it needs two regexes and a recursive tree walk, not a generated parser. Supported: dotted node references (${a.b.c}), resolver calls with typed literal arguments (${globals:x}, ${oc.env:VAR,default}), string concatenation ("${a}/${b}"), type preservation when a value is exactly one interpolation, and \${...} escaping. Resolution is eager with a cycle guard. Strings without ${ cost one substring check.

Not supported, by design: nested interpolations (${a.${b}}) and quoted resolver arguments containing commas.

Bug-for-bug fidelity

Matching OCL means matching OmegaConf's quirks, not YAML-the-spec. The behaviors deepyaml reproduces deliberately — each derived by reading Kedro's and OmegaConf's source, and each pinned by a test:

• YAML dialect: YAML 1.1 typing (no → False, 12:30 → 750), plus OmegaConf's extra float resolver (1e3 → 1000.0), minus the timestamp resolver (dates stay strings). Duplicate keys in one mapping raise rather than last-wins.
• Environment merging is destructive by default — top-level keys are replaced, not deep- merged; merge_strategy={"key": "soft"} opts into deep merge.
• Interpolation scope is per-environment — conf/local cannot reference a conf/base key.
• oc.env is credentials-only, and resolved per file, before files are merged.
• Duplicate detection compares full dotted leaf paths for parameters, top-level keys everywhere else.
• Missing-config semantics, including returning {} when run_env == base_env and raising MissingConfigException otherwise, so the framework's optional parameters/credentials handling keeps working.
• _-prefixed templating keys stripped from every key except parameters; runtime params deep-merged into parameters per environment, pre-resolution, unstripped.

Known divergences

Published in the README, not discovered by users:

• Local filesystems only. No fsspec-backed conf sources.
• Error types. MissingConfigException and ValueError (duplicate keys) match Kedro's; parse and interpolation failures raise deepyaml's own DeepYAMLError subclasses with equivalent messages.
• JSON is parsed by json, not routed through the YAML parser as OmegaConf does.
• Resolver registry. Resolvers registered via OmegaConf.register_new_resolver are not visible; use deepyaml.register_resolver or custom_resolvers.
• Files within one directory merge in sorted order (Kedro's order is unspecified); observable only for overlapping _-prefixed keys.

Benchmarks

End-to-end load of all default keys (globals, catalog, parameters, credentials), median of 5 runs, outputs verified identical before timing:

macOS 13.6 (arm64), CPython 3.10.18, kedro 1.6.0, omegaconf 2.3.1, PyYAML 6.0.3 (libyaml), 2026-09-09. The gap widens with interpolation density, which is the signature you'd expect if antlr4 is the term being removed — consistent with the profiles in #4322.

Caveat, stated up front: these are my generated config trees on my machine with my harness. That is exactly what I'd like to fix — see Testing.

What about OmegaConf 2.4?

The obvious objection is that OmegaConf 2.4 removes the antlr4 dependency, so perhaps this solves itself. It doesn't: 2.4 vendors antlr4 into the package (omegaconf/vendor/antlr4, ~900 KB) rather than replacing it. The grammar and the per-interpolation parsing cost are unchanged.

Measured, same machine and same config trees, against omegaconf==2.4.0.dev15:

2.4 is roughly 10% faster and remains 25–28× slower than deepyaml. So the vendoring fixes installability (including the Pyodide blocker above) without addressing the performance question.

Separately, and worth flagging for whoever picks up the 2.4 upgrade: kedro 1.6.0 against omegaconf 2.4 emits register_new_resolver() is deprecated warnings from kedro/config/omegaconf_config.py:592 and :697. That migration is coming regardless of this proposal.

Alternatives considered

The honest cost of (D) is maintenance, not performance: a reimplementation must track OCL's behavior as it changes. The mitigation is that the equivalence suite is executable and runs against the real OmegaConfigLoader, so drift shows up as a test failure rather than a user's bug report.

Testing

153 tests, all passing against kedro 1.6.0 / omegaconf 2.3.1 (re-verified 2026-09-09, the day 1.6.0 shipped):

• 28 equivalence tests — the load-bearing ones. Each builds a config scenario, runs it through both the real OmegaConfigLoader and DeepYAMLConfigLoader, and asserts type-identical output. This is the executable spec, and the drift detector.
• 14 adapter tests — AbstractConfigLoader contract, exception translation, globals.
• 4 kedro run integration tests — a real project run end-to-end through CONFIG_LOADER_CLASS.
• 107 unit tests across parsing, merging, interpolation, and loader semantics.
• The benchmark harness itself asserts identical outputs before it times anything.

Measured against Kedro's own benchmark suite

Rather than ask you to trust my harness, I ran kedro_benchmarks/benchmark_ocl.py — Kedro's asv suite, with the config shapes the team chose (a 1,000-entry base catalog and a 1,000-entry local overlay). I drove the benchmark bodies and fixtures directly rather than through asv, so there is no env-matrix or commit-checkout machinery, but the measured work is what the suite measures. Fresh loader per sample, median of 3.

Every key produced identical output through both loaders on these fixtures (catalog, parameters, globals, across both the basic and interpolated fixture sets).

TimeOmegaConfigLoader:

TimeOmegaConfigLoaderAdvanced:

The multiples are lower than my own harness reports (12–19× rather than 25–31×) because these benchmarks load a single key at a time rather than a whole conf tree. I'd rather quote these: they're your shapes, not mine.

Remaining questions for the session

1. Does the asv OCL suite miss shapes you care about (versioned datasets, deep globals.yml, many small files vs. few large)? If so I'll add them.
2. Should I open a PR adding a TimeDeepYAMLConfigLoader to kedro_benchmarks/, so this runs in the real harness rather than my transcription of it?
3. Is there a real project — internal or community — whose conf/ I could run the equivalence suite against? Passing on synthetic scenarios is much weaker evidence than passing on someone's actual catalog.

Rollout strategy

Fully backward compatible; nothing to migrate. deepyaml is a separate package, and adoption today is two lines in settings.py:

from deepyaml.kedro import DeepYAMLConfigLoader

CONFIG_LOADER_CLASS = DeepYAMLConfigLoader

CONFIG_LOADER_ARGS (base_env, default_run_env, config_patterns, custom_resolvers, merge_strategy) work unchanged. If a project hits a divergence, reverting is deleting those two lines.

This means no decision is required from Kedro for the approach to be useful — which is why I'm bringing it as a design discussion rather than a KEP. If it proves out, the natural escalation path is the KEP process, for something like a pluggable resolution backend behind OCL.

Future iterations

• fsspec-backed conf sources — the largest functional gap vs. OCL, and the one most likely to block real adoption.
• Kedro asv integration — as above, pending agreement on the yardstick.
• YAML 1.2 backend — the parsing layer is pluggable; a saphyr-based parser is both spec-correct and measurably faster than libyaml, as an opt-in.
• Nested interpolation (${a.${b}}) if anyone's configs actually use it — I have not seen it in Kedro projects, and would rather not add grammar without a real use case.
• Optional Rust hot path, validated against the pure-Python reference. Explicitly not needed for the performance claim above.