Baike.dev
Log in
Back to tool/Back to issues
#32217·dagster

Dynamic partitions + managed_python_loggers causes dagster to hang

Author: OrenLedermanCreated Sep 18, 2025Updated Sep 18, 2026
Labelstype: bugstale

What's the issue?

When using dynamic partitions with managed_python_loggers configured in dagster.yaml and SQLite storage (the default), Dagster hangs after the asset yield the output. Here's the last log it shows:

Yielded output "result" of type "[xxxx]". (Type check passed).

py-spy dump provides the following stack:

    _connect (dagster/_core/storage/event_log/sqlite/sqlite_event_log.py:225)
    __enter__ (contextlib.py:135)
    store_event (dagster/_core/storage/event_log/sqlite/sqlite_event_log.py:257)
    handle_new_event (dagster/_core/instance/__init__.py:2463)
    emit (dagster/_core/instance/__init__.py:261)
    handle (logging/__init__.py:968)
    emit (dagster/_core/log_manager.py:268)
    handle (logging/__init__.py:968)
    callHandlers (logging/__init__.py:1696)
    handle (logging/__init__.py:1634)
    _log (logging/__init__.py:1624)
    info (logging/__init__.py:1477)
    __init__ (alembic/runtime/migration.py:207)
    configure (alembic/runtime/migration.py:273)
    check_alembic_revision (dagster/_core/storage/sql.py:73)
    _initdb (dagster/_core/storage/event_log/sqlite/sqlite_event_log.py:179)
    _connect (dagster/_core/storage/event_log/sqlite/sqlite_event_log.py:232)
    __enter__ (contextlib.py:135)
    has_dynamic_partition (dagster/_core/storage/event_log/sql_event_log.py:2098)
    has_dynamic_partition (dagster/_core/instance/__init__.py:2385)
    inner (dagster/_utils/__init__.py:691)
    has_partition_key (dagster/_core/definitions/partition.py:559)
    get_partition_keys_in_range (dagster/_core/definitions/partition.py:176)
    asset_partition_keys (dagster/_core/execution/context/output.py:489)
    _get_paths_for_partitions (dagster/_core/storage/upath_io_manager.py:242)
    handle_output (dagster/_core/storage/upath_io_manager.py:431)
    _gen_fn (dagster/_core/execution/plan/execute_step.py:767)
    iterate_with_context (dagster/_utils/__init__.py:480)
    _store_output (dagster/_core/execution/plan/execute_step.py:776)
    _type_check_and_store_output (dagster/_core/execution/plan/execute_step.py:554)
    core_dagster_event_sequence_for_step (dagster/_core/execution/plan/execute_step.py:508)
    dagster_event_sequence_for_step (dagster/_core/execution/plan/execute_plan.py:245)
    inner_plan_execution_iterator (dagster/_core/execution/plan/execute_plan.py:87)
    __iter__ (dagster/_core/execution/api.py:869)
    execute (dagster/_core/executor/multiprocess.py:97)
    _execute_command_in_child_process (dagster/_core/executor/child_process_executor.py:81)
    run (multiprocessing/process.py:108)
    _bootstrap (multiprocessing/process.py:314)
    _main (multiprocessing/spawn.py:129)
    spawn_main (multiprocessing/spawn.py:116)
    <module> (<string>:1)
Thread 0x16E057000 (idle): "QueueFeederThread"
    wait (threading.py:320)
    _feed (multiprocessing/queues.py:231)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)
Thread 0x16F063000 (idle): "termination-handler"
    wait (multiprocessing/synchronize.py:261)
    wait (multiprocessing/synchronize.py:349)
    _termination_handler (dagster/_utils/__init__.py:433)
    run (threading.py:953)
    _bootstrap_inner (threading.py:1016)
    _bootstrap (threading.py:973)

Based on what I can tell, this is what's happening: The issue occurs during the I/O manager's output handling phase:

  1. Asset calls has_dynamic_partition() which requires a SQLite database connection
  2. _connect() calls check_alembic_revision() to verify database schema
  3. Alembic logs an INFO message: "Context impl SQLiteImpl."
  4. Dagster captures this log because of managed_python_loggers: [root] configuration
  5. Dagster tries to store_event() in the SQLite event log database
  6. store_event() needs a database connection, so calls _connect() again and gets stuck

If I remove the following lines from my dagster.yaml file, the issue is resolved:

python_logs:
   managed_python_loggers:
     - root

What did you expect to happen?

I expect to be able to turn on detailed logs and use dynamic partitions :)

How to reproduce?

Create an asset with dynamic partitions. Include the following in your dagster.yaml file:

python_logs:
  managed_python_loggers:
    - root

Make sure to use the default sqlite backend

Dagster version

1.9.13

Deployment type

Local

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a ! We factor engagement into prioritization.

Source: dagster-io/dagster

View original on GitHubView discussion on GitHub