[Spike] Refactor Kedro Pipeline Discovery into an Explicit Registry Architecture

Description

As an outcome to the spike work - https://github.com/kedro-org/kedro/issues/5406

Kedro’s current pipeline discovery and loading flow is difficult to reason about because several concerns are coupled together inside kedro/framework/project/__init__.py.

Today, pipeline discovery depends on a mixture of:

• the global pipelines singleton,
• _ProjectPipelines,
• find_pipelines(),
• _load_data(),
• register_pipelines() lookup,
• pipeline_registry.py,
• lazy dict-proxy behavior,
• and the _requested_pipelines side channel used by CLI commands. (after #5549 )

This makes the loading order fragile. Commands that need only pipeline names, a single pipeline, or the default pipeline set may still need to carefully manage global state before the first access to pipelines. The current flow also makes future dependency-free or import-light workflows harder to implement.

This ticket proposes refactoring the pipeline discovery flow into an explicit registry architecture, centered around a PipelineRegistry abstraction and a default AutoDiscoveryRegistry.

This may be a staged non-breaking migration initially, with a potential breaking cleanup in a future major release.

Context

Kedro has three distinct levels of pipeline laziness:

Layer 1 — Pipeline names
Zero pipeline imports. Uses directory scanning or pyproject.toml metadata only.

Layer 2 — Pipeline modules
Imports only the requested pipeline module, such as my_project.pipelines.data_science.

Layer 3 — Node functions
Imports heavy node dependencies only when the node runs. This remains TBD.

The current _requested_pipelines approach partially addresses Layer 1 and Layer 2, but it does so through implicit global state. CLI commands must call pipelines.set_requested(...) before anything accesses the global pipelines mapping. This works, but the ordering requirement is subtle and easy to break.

The broader refactor would make the loading sequence explicit per command. Instead of commands mutating global state and relying on later discovery behavior, each command would ask the registry for exactly what it needs:

• list pipeline names,
• load a single named pipeline,
• load the configured default pipeline set,
• or load all discoverable pipelines.

This also allows kedro/framework/project/__init__.py to return to its core responsibilities: settings, logging, package metadata, and project bootstrap.

Proposed Architecture

Introduce a new registry layer under:

kedro/pipeline/registry.py

This module would contain:

class PipelineRegistry(Protocol):
    def list(self) -> list[str]:
        ...

    def load(self, name: str) -> Pipeline:
        ...

    @property
    def default_names(self) -> list[str]:
        ...

    @property
    def default(self) -> Pipeline:
        ...

and a default implementation:

class AutoDiscoveryRegistry:
    ...

The registry becomes the single owner of pipeline discovery and loading.

New File Topology

kedro/
  pipeline/
    registry.py        # PipelineRegistry Protocol + AutoDiscoveryRegistry

  framework/
    project/
      __init__.py      # settings, LOGGING, PACKAGE_NAME, configure_project() only
      _settings.py     # _ProjectSettings / Dynaconf wrapper
      _logging.py      # _ProjectLogging

    session/
      session.py       # no set_requested(), no pipelines global dependency

    cli/
      project.py       # explicit load sequence per command

What Moves Out of framework/project/__init__.py

The following pipeline-related pieces would be removed from kedro/framework/project/__init__.py and replaced by the registry layer:

After this refactor, framework/project/__init__.py would primarily own:

settings
LOGGING
PACKAGE_NAME
configure_project()

CLI-Demand-Driven Loading

Each CLI command should define its pipeline demand profile explicitly.

This makes behavior clear and testable.

For example, kedro registry list should only perform Layer 1 discovery. It should not import pipeline modules or node dependencies.

kedro registry describe X should import only pipeline X.

kedro run --pipeline X should load config and catalog, then import and run only pipeline X.

Example run Flow

# kedro/framework/cli/project.py

@click.command()
@click.option("--pipeline", multiple=True)
def run(pipeline, ...):
    registry = _get_registry()                         # reads pyproject.toml; zero pipeline imports
    names = pipeline or registry.default_names          # Layer 1

    with KedroSession.create(...) as session:
        catalog = session.load_catalog()               # config + catalog only

        for name in names:
            pl = registry.load(name)                   # Layer 2; one pipeline at a time
            runner.run(pl, catalog, ...)

This removes the need for:

pipelines.set_requested(...)

and removes the temporal ordering issue where the filter must be set before first access to the global pipelines object.

AutoDiscoveryRegistry

A default registry implementation could look like this:

# kedro/pipeline/registry.py

class AutoDiscoveryRegistry:
    def __init__(self, package_name: str, config: PipelineConfig):
        self._package = package_name
        self._exclude = set(config.exclude or [])
        self._default_names = config.default
        self._cache: dict[str, Pipeline] = {}

    def list(self) -> list[str]:
        """Layer 1 — directory scan only, zero pipeline imports."""
        pkg = importlib.resources.files(f"{self._package}.pipelines")

        return [
            d.name
            for d in pkg.iterdir()
            if d.is_dir()
            and not d.name.startswith("_")
            and d.name not in self._exclude
        ]

    def load(self, name: str) -> Pipeline:
        """Layer 2 — import one pipeline module on demand."""
        if name not in self._cache:
            mod = importlib.import_module(f"{self._package}.pipelines.{name}")
            self._cache[name] = mod.create_pipeline()

        return self._cache[name]

    @property
    def default_names(self) -> list[str]:
        return self._default_names or self.list()

    @property
    def default(self) -> Pipeline:
        return sum(self.load(name) for name in self.default_names) or Pipeline([])

Open design questions:

• Should load(name) validate that name appears in list() before importing?
• Should the registry support aliases such as __default__?
• Should default be a real pipeline name, a computed property, or both?
• Should registry caching be per command invocation, per session, or global?
• How should missing create_pipeline() be reported?
• How should discovery behave when my_project.pipelines does not exist?
• Should namespace packages be supported via importlib.resources.files()?

pyproject.toml Configuration

Add a new optional configuration section:

[tool.kedro.pipelines]
exclude = ["experimental", "legacy"]
default = ["data_processing", "data_science"]

exclude

Directories listed in exclude are ignored by auto-discovery.

They should be absent from:

kedro registry list
kedro run

and should not be loaded unless a custom registry chooses to expose them.

default

The default list defines what kedro run executes when no --pipeline flag is provided.

If default is absent, Kedro combines all discovered, non-excluded pipelines.

This makes default pipeline composition declarative and avoids requiring every project to maintain pipeline_registry.py solely to define the default run set.

Programmatic Escape Hatch

Add a settings hook:

# settings.py

PIPELINE_REGISTRY_CLASS = "my_project.pipeline_registry.CustomRegistry"

or, depending on existing settings conventions:

from my_project.pipeline_registry import CustomRegistry

PIPELINE_REGISTRY_CLASS = CustomRegistry

This supports projects that cannot be represented by directory scanning and pyproject.toml, including:

• namespaced pipelines,
• dynamically generated pipelines,
• plugin-provided pipelines,
• tenant-specific or environment-specific pipeline composition,
• advanced default pipeline logic,
• non-standard project layouts.

The custom registry should implement the same PipelineRegistry protocol.

Migration Path

Phase 1 — Introduce registry abstraction

Add:

PipelineRegistry
AutoDiscoveryRegistry
PIPELINE_REGISTRY_CLASS
[tool.kedro.pipelines]

Default new projects can use AutoDiscoveryRegistry.

Existing projects with pipeline_registry.py continue to work unchanged.

Breaking: no.

Phase 2 — Bridge existing pipeline_registry.py

If pipeline_registry.py exists, Kedro continues to use it but emits a deprecation warning.

The warning should explain how to migrate to either:

[tool.kedro.pipelines]
default = [...]
exclude = [...]

or:

PIPELINE_REGISTRY_CLASS = CustomRegistry

Breaking: no.

Phase 3 — Prefer auto-discovery when pipeline_registry.py is absent

If pipeline_registry.py is absent, Kedro uses:

AutoDiscoveryRegistry
+ pyproject.toml
+ directory scan

This allows projects to avoid creating a registry file at all.

Breaking: no.

Phase 4 — Remove legacy registry path in a major release

Remove support for:

pipeline_registry.py
register_pipelines()
find_pipelines()
_ProjectPipelines
global pipelines singleton
_requested_pipelines

Breaking: yes.

This should only happen in a major version release after deprecation warnings and migration documentation have been available for at least one release cycle.

Compatibility Considerations

Non-breaking path

This can begin as a non-breaking change if Kedro keeps supporting:

def register_pipelines():
    ...

and the global pipelines behavior during the migration period.

Breaking path

The final cleanup is likely breaking because many projects may import or rely on:

from kedro.framework.project import pipelines
from kedro.framework.project import find_pipelines

or define:

my_project/pipeline_registry.py

with:

def register_pipelines():
    ...

Backward compatibility bridge

During migration, Kedro could provide an adapter registry:

class LegacyPipelineRegistry:
    def __init__(self, register_pipelines: Callable[[], dict[str, Pipeline]]):
        self._register_pipelines = register_pipelines
        self._cache: dict[str, Pipeline] | None = None

    def list(self) -> list[str]:
        return list(self._load_all())

    def load(self, name: str) -> Pipeline:
        return self._load_all()[name]

    @property
    def default(self) -> Pipeline:
        return self._load_all()["__default__"]

    def _load_all(self) -> dict[str, Pipeline]:
        if self._cache is None:
            self._cache = self._register_pipelines()
        return self._cache

This preserves compatibility but does not provide true Layer 1 or Layer 2 laziness for legacy registries.

Testing Requirements

This proposal needs a spike and test coverage before implementation.

Suggested test areas:

Layer 1 discovery

• registry.list() performs directory scanning only.
• registry.list() does not import pipeline modules.
• Excluded directories are omitted.
• Private directories such as _utils are omitted.
• Missing pipelines/ package is handled with a useful error or empty result, depending on desired behavior.
• Namespace packages are handled or explicitly unsupported.

Layer 2 loading

• registry.load("data_science") imports only my_project.pipelines.data_science.
• Loading one pipeline does not import sibling pipelines.
• Loaded pipelines are cached.
• Missing pipeline names produce clear errors.
• Missing create_pipeline() produces clear errors.
• Exceptions raised during pipeline import preserve useful traceback context.

Default pipeline behavior

• If [tool.kedro.pipelines].default is set, kedro run loads only those names.
• If default is absent, kedro run loads all discovered, non-excluded pipelines.
• If default contains an excluded pipeline, Kedro raises a clear configuration error.
• If default contains a missing pipeline, Kedro raises a clear configuration error.
• Default composition order follows the order in pyproject.toml.

CLI demand profiles

• kedro registry list does not import pipeline modules.
• kedro registry describe X imports only X.
• kedro run --pipeline X imports only X.
• kedro run --pipeline X --pipeline Y imports only X and Y.
• kedro run imports only configured default pipelines.
• kedro catalog list does not import pipeline modules.
• kedro catalog list -p X validates or references only pipeline X without loading unrelated modules, depending on final design.

Legacy compatibility

• Existing pipeline_registry.py projects continue to work during non-breaking phases.
• Existing find_pipelines() usage continues to work until formal deprecation/removal.
• Existing imports from kedro.framework.project import pipelines either continue to work or emit a deprecation warning during migration.
• Custom registries can be configured through PIPELINE_REGISTRY_CLASS.

Error and warning behavior

• Deprecation warning is emitted when pipeline_registry.py is used.
• Warning includes migration guidance.
• Invalid pyproject.toml pipeline config produces actionable errors.
• CLI error messages distinguish between discovery errors, import errors, and missing pipeline names.

Possible Alternatives

Alternative 1 — Keep _requested_pipelines and improve coverage

Kedro could continue extending the current side-channel approach by adding more set_requested() calls across CLI commands.

This is lower risk but preserves the global-state and temporal-ordering issues. The discovery model remains hard to understand and the pipeline logic stays concentrated in framework/project/__init__.py.

Alternative 2 — Only move code out of __init__.py

Kedro could refactor the existing behavior into smaller modules without changing the architecture.

This would improve maintainability but would not solve the core issue that commands interact with a global lazy mapping rather than an explicit registry.

Alternative 3 — Add only pyproject.toml defaults

Kedro could add [tool.kedro.pipelines] default = [...] while preserving pipeline_registry.py and find_pipelines() as the primary architecture.

This helps with declarative default composition but does not create a clean demand-driven loading model.

Alternative 4 — Implement Layer 3 laziness first

Kedro could focus on lazy node function imports before refactoring discovery.

Layer 3 is valuable, especially for heavy dependencies inside node modules, but it is a larger design problem. Refactoring Layer 1 and Layer 2 first gives Kedro a clearer foundation for later Layer 3 work.

Acceptance Criteria

• A PipelineRegistry abstraction is introduced.
• AutoDiscoveryRegistry supports zero-import pipeline name discovery.
• AutoDiscoveryRegistry.load(name) imports only the requested pipeline module.
• [tool.kedro.pipelines].exclude is supported.
• [tool.kedro.pipelines].default is supported.
• PIPELINE_REGISTRY_CLASS is supported as a programmatic escape hatch.
• CLI commands use explicit demand-driven loading instead of mutating _requested_pipelines.
• kedro registry list does not import pipeline modules.
• kedro registry describe X imports only pipeline X.
• kedro run --pipeline X imports only pipeline X.
• kedro run imports only the configured default pipeline set.
• Existing pipeline_registry.py projects remain supported during the non-breaking migration phases.
• Deprecation warnings are emitted for legacy registry usage during the migration window.
• The final breaking-removal phase is documented separately for a future major release.
• Test coverage verifies import behavior, default behavior, CLI demand profiles, legacy compatibility, and error handling.