Behavior of `output_required` and materialization of downstream assets is unexpected
What's the issue?
From community member Mykola Palamarchuk.
This part of documentation says that you can return None , so downstream assets will be skipped. But it doesn't work as described (latest Dagster 1.8.11).
from dagster import asset, Definitions, define_asset_job
@asset(output_required=False)
def x():
return None
@asset
def plus1(x): # should never materialize
return x+1
the_job = define_asset_job(
name="the_job",
selection=['*plus1'],
)
defs = Definitions(
assets=[x, plus1],
jobs=[the_job]
)Materializing the_job leads to plus1 asset materialization with error unsupported operand type(s) for +: 'NoneType' and 'int' . There is also a warning that says:
Value "None" returned for non-required output "result" of op "x". This value will be passed to downstream ops. For conditional execution, results must be yielded: https://docs.dagster.io/concepts/ops-jobs-graphs/graphs#with-conditional-branchingSee original discussion in community Slack:
What did you expect to happen?
The expected behavior aligns with the docs of output_required.
Whether the decorated function will always materialize an asset. Defaults to True. If False, the function can return None, which will not be materialized to storage and will halt execution of downstream assets.
How to reproduce?
See example.
Dagster version
1.8.11
Deployment type
Local
Deployment details
No response
Additional information
No response
Message from the maintainers
Impacted by this issue? Give it a ! We factor engagement into prioritization. By submitting this issue, you agree to follow Dagster's Code of Conduct.
Source: dagster-io/dagster