Pre-flight validation for node groupings before deployment
Description
Add a read only check that takes a pipeline, a grouping from Pipeline.group_nodes_by() and the catalog, and reports problems that would make the grouped pipeline fail once each group runs as a separate task on a deployment platform.
Context
When a project is deployed, each group of nodes runs in its own container. A dataset that only exists in memory cannot cross from one container to the next, and today nothing reports this before deployment. Users find out when the platform run fails. "Validating the groups" is one of the three untouched recommendations from #3094, and it was agreed as the highest value core item in the deployment workstream meeting ,https://github.com/kedro-org/kedro/issues/2058#issuecomment-5715275626.
The nearest precedent in core is SharedMemoryDataCatalog.validate_catalog(), which rejects in-memory data crossing a process boundary for ParallelRunner.
Scope:
- Report datasets produced in one group and consumed in another that are ephemeral: no catalog entry, or an entry that resolves to MemoryDataset or another
_EPHEMERALdataset. Dataset factory patterns must be resolved first. - Report cycles between groups. _group_by_namespace can produce them when a node outside a namespace sits between two nodes inside it.
- Warn on boundary datasets with a local filesystem path, since local disk is not shared between containers either.
- List single-node groups, which is what un-namespaced nodes become, as information rather than failure.
- Return a structured, JSON-safe result in the same style as ValidationResult from #5680 (status, list of issues, to_dict(), truthiness).
- Unit tests on spaceflights-style pipelines, docs section in docs/deploy/nodes_grouping.md.
Open questions:
Where it lives: kedro.pipeline, kedro.inspection, or a new module? Whether it should also accept a ProjectSnapshot so tools can run it without holding live pipeline and catalog objects? CLI entry point for deployment a command matters more, because CI jobs and AI skills call shell commands. Could be a follow-up?
Source: kedro-org/kedro