由于 GX 上下文数据源出错,无法同时运行多个 Great Expectations 检查
作者: monishwaran07创建于 2026年9月16日更新于 2026年9月16日
标签bug
Describe the bug Great Expectations validation is failing with the following error: ResourceFreshnessAggregateError: Could not find datasource '86e311562041' The error occurs while creating the ValidationDefinition from the BatchDefinition. The datasource referenced by the BatchDefinition cannot be found, causing the GX validation to fail. The issue is observed in the following cases: 1. Checks within the same table: The initial checks are executed successfully, but subsequent checks are not executed after the datasource error occurs. 2. Different tables running in parallel: When checks for different tables are executed in parallel, some checks are executed successfully while the remaining checks fail with the datasource error. 3. Different APIs running in parallel: The datasource issue is also observed when different APIs execute checks in parallel. Need to investigate why the datasource referenced by the BatchDefinition is not available when the ValidationDefinition is created. [2026-09-16 12:36:35] - INFO - a383802a-405c-494d-a9d2-0fb34041fdeb - Creating data quality engine for database type: SNOWFLAKE [2026-09-16 12:36:35] - INFO - a383802a-405c-494d-a9d2-0fb34041fdeb - Snowflake secret key fetched successfully [2026-09-16 12:36:35] - INFO - a383802a-405c-494d-a9d2-0fb34041fdeb - Snowflake private key encoded successfully [2026-09-16 12:36:35] - INFO - a383802a-405c-494d-a9d2-0fb34041fdeb - api: GX table=2, SQL table=11 (w=8), GX col=10, SQL col=0 (w=0) [2026-09-16 12:36:35] - INFO - a383802a-405c-494d-a9d2-0fb34041fdeb - GX Thread: Starting for table [2026-09-16 12:36:35] - INFO - a383802a-405c-494d-a9d2-0fb34041fdeb - SQL Table Pool: Running 11 checks on table with 8 worker(s) 2026-09-16 12:36:35,443 - INFO - Created temporary directory 'C:\Users\AppData\Local\Temp\tmpqmaj6zqe' for ephemeral docs site [2026-09-16 12:36:38] - INFO - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - GX ephemeral context created for data quality checks [2026-09-16 12:36:39] - INFO - a383802a-405c-494d-a9d2-0fb34041fdeb - GX ephemeral context created for data quality checks [2026-09-16 12:36:39] - INFO - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - SQL Column Pool: Completed [2026-09-16 12:36:39] - INFO - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - Thread 'sql_column' completed [2026-09-16 12:36:40] - ERROR - D:\home\checks_definition.py - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - GX Thread failed: Could not find datasource '86e311562041' Traceback (most recent call last): File "home\checks_definition.py", line 1196, in run_gx_checks validation_definition = gx_context.validation_definitions.add( ValidationDefinition(name=_uid(), data=batch_def, suite=suite) ) File "D:\abc\Lib\site-packages\great_expectations\core\factory\validation_definition_factory.py", line 43, in add self._store.add(key=key, value=validation) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^ File
内容来源: fivetran/great_expectations