Dask deployment guide: custom runner recipe is broken

repro_dask_doc.py

Description

The "Create a custom runner" recipe on the Dask page no longer runs against a current Kedro. Three separate parts are broken. I have a fix ready for all three and would be glad to open the PR if you are happy with the approach.

1. The sample overrides API that no longer exists on AbstractRunner

The recipe asks the reader to copy a DaskRunner that defines create_default_dataset() and run_only_missing(). Neither method exists any more:

>>> from kedro.runner import AbstractRunner
>>> for name in ("run_only_missing", "create_default_dataset", "create_default_dataset"):
...     print(name, hasattr(AbstractRunner, name))
run_only_missing False
create_default_dataset False

run_only_missing() also calls three methods that are gone from DataCatalog / CatalogProtocol:

>>> from kedro.io import DataCatalog, CatalogProtocol
>>> for name in ("list", "add", "shallow_copy"):
...     print(name, hasattr(DataCatalog, name), hasattr(CatalogProtocol, name))
list False False
add False False
shallow_copy False False

So if anything ever reaches that method it raises AttributeError immediately; otherwise it is dead code that contradicts the base class. _DaskDataset is only referenced by those two methods, so it disappears with them.

2. The cli.py snippet is not valid Python

Under "Update CLI implementation" the page prints:

def run(tag, env, ...):
    """Run the pipeline."""
    runner = runner or "SequentialRunner"

    tags = tuple(tags)
    node_names = tuple(node_names)
    ...

ast.parse on that snippet gives SyntaxError: invalid syntax on line 1, and the body refers to runner and params, which are not parameters of the function.

The section also appears to be unnecessary. kedro run already resolves --runner with load_obj(runner or "SequentialRunner", "kedro.runner") and forwards --runner-params to the constructor (kedro/framework/cli/project.py), so a custom runner can be instantiated without overriding any project command:

kedro run --runner=kedro_tutorial.runner.DaskRunner \
  --runner-params=client_args.address=127.0.0.1:8786

That also replaces the conf/dask/parameters.yml step further down the page, which only existed to feed client_args through the custom cli.py.

3. The catalogue is serialised to the workers, which the guide does not mention

_run_node receives catalog as an argument, so client.submit() serialises it and each worker gets its own copy. The copy is taken when the node is submitted, before the upstream node has written anything, so an in-memory dataset produced by one node is not visible to a node running in another worker. This is most likely the underlying cause of #4657 and #5233; readers follow the page, get a runner that appears to work, and then see DatasetError from whichever node consumed the previous node's output.

Context

Issues #4657, #5233 and #5732 all report that following this page does not work, and #4657 explicitly asks for "the files that are described, or to include it as a standard runner". This issue is the smaller half of that request: keep the recipe as a recipe, but make it correct.

Proposed fix

A docs-only change to docs/deploy/supported-platforms/dask.md:

1. Drop create_default_dataset(), run_only_missing() and _DaskDataset from the sample, so it only uses the current AbstractRunner / CatalogProtocol API.
2. Replace the cli.py section with --runner-params, including the dotted-key form used to pass client_args, and drop the now-redundant conf/dask/parameters.yml step.
3. Make __del__ tolerant of Client.current() returning None.
4. Add a warning that datasets are not shared between workers, with a pointer to persistent datasets.

After the change every Python block on the page parses, and the sample no longer references removed API.

I am happy to open this as a PR, or to hand it over if you would rather fix it in-house.