Baike.dev
Log in
Back to tool/Back to issues
#12209·great_expectations

Unable to run multiple Great Expectations checks concurrently due to GX Context datasource error

Author: monishwaran07Created Sep 16, 2026Updated Sep 16, 2026
Labelsbug

Describe the bug

Great Expectations validation is failing with the following error:

ResourceFreshnessAggregateError: Could not find datasource '86e311562041'

The error occurs while creating the ValidationDefinition from the BatchDefinition. The datasource referenced by the BatchDefinition cannot be found, causing the GX validation to fail.

The issue is observed in the following cases: 1. Checks within the same table: The initial checks are executed successfully, but subsequent checks are not executed after the datasource error occurs.

2. Different tables running in parallel: When checks for different tables are executed in parallel, some checks are executed successfully while the remaining checks fail with the datasource error.

3. Different APIs running in parallel: The datasource issue is also observed when different APIs execute checks in parallel.

Need to investigate why the datasource referenced by the BatchDefinition is not available when the ValidationDefinition is created.

[2026-09-16 12:36:35]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - Creating data quality engine for database type: SNOWFLAKE
[2026-09-16 12:36:35]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - Snowflake secret key fetched successfully  
[2026-09-16 12:36:35]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - Snowflake private key encoded successfully  
[2026-09-16 12:36:35]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - api: GX table=2, SQL table=11 (w=8), GX col=10, SQL col=0 (w=0)
[2026-09-16 12:36:35]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - GX Thread: Starting for table  
[2026-09-16 12:36:35]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - SQL Table Pool: Running 11 checks on  table with 8 worker(s)
2026-09-16 12:36:35,443 - INFO - Created temporary directory 'C:\Users\AppData\Local\Temp\tmpqmaj6zqe' for ephemeral docs site
[2026-09-16 12:36:38]  - INFO  - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - GX ephemeral context created for data quality checks
[2026-09-16 12:36:39]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - GX ephemeral context created for data quality checks
[2026-09-16 12:36:39]  - INFO  - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - SQL Column Pool: Completed  
[2026-09-16 12:36:39]  - INFO  - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - Thread 'sql_column' completed  
[2026-09-16 12:36:40]  - ERROR - D:\home\checks_definition.py - d4feeb03-eaa8-4069-b798-0ae89ec2b394 -  GX Thread failed:
        Could not find datasource '86e311562041'
Traceback (most recent call last):
  File "home\checks_definition.py", line 1196, in run_gx_checks
    validation_definition = gx_context.validation_definitions.add(
        ValidationDefinition(name=_uid(), data=batch_def, suite=suite)
    )
  File "D:\abc\Lib\site-packages\great_expectations\core\factory\validation_definition_factory.py", line 43, in add    
    self._store.add(key=key, value=validation)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\req\Lib\site-packages\great_expectations\data_context\store\store.py", line 286, in add
    return self._add(key=key, value=value, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\req\Lib\site-packages\great_expectations\data_context\store\validation_definition_store.py", line 100, in _add
    return super()._add(key=key, value=value, **kwargs)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\req\Lib\site-packages\great_expectations\data_context\store\store.py", line 290, in _add
    output = self._store_backend.add(self.key_to_tuple(key), self.serialize(value), **kwargs)
                                                             ~~~~~~~~~~~~~~^^^^^^^
  File "D:\req\Lib\site-packages\great_expectations\data_context\store\validation_definition_store.py", line 77, in serialize
    output = value.json(models_as_dict=False, indent=2, sort_keys=True)
  File "D:\req\Lib\site-packages\pydantic\v1\main.py", line 510, in json
    return self.__config__.json_dumps(data, default=encoder, **dumps_kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\UsersAppData\Local\Programs\Python\Python313\Lib\json\__init__.py", line 242, in dumps
    **kw).encode(obj)
          ~~~~~~^^^^^
  File "C:\Users\AppData\Local\Programs\Python\Python313\Lib\json\encoder.py", line 202, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Users\AppData\Local\Programs\Python\Python313\Lib\json\encoder.py", line 263, in iterencode
    return _iterencode(o, 0)
  File "D:\req\Lib\site-packages\pydantic\v1\json.py", line 101, in custom_pydantic_encoder
    return encoder(obj)
  File "D:\req\Lib\site-packages\great_expectations\core\validation_definition.py", line 114, in <lambda>
    BatchDefinition: lambda b: b.identifier_bundle(),
                               ~~~~~~~~~~~~~~~~~~~^^
  File "D:\req\Lib\site-packages\great_expectations\core\batch_definition.py", line 170, in identifier_bundle
    diagnostics.raise_for_error()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "D:\req\Lib\site-packages\great_expectations\core\freshness_diagnostics.py", line 49, in raise_for_error        
    raise self.raise_for_error_class(errors=self.errors)
great_expectations.exceptions.resource_freshness.ResourceFreshnessAggregateError:
        Could not find datasource '86e311562041'
[2026-09-16 12:36:40]  - INFO  - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - Thread 'gx' completed  
[2026-09-16 12:36:40]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - GX Thread: Running validation with result_format
Calculating Metrics:   0%|                                                                         | 0/38 [00:00<?, ?it/s][2026-09-16 12:36:46]  - INFO  - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - SQL Table Pool: Completed — 6 checks
[2026-09-16 12:36:46]  - INFO  - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - Thread 'sql_table' completed  
[2026-09-16 12:36:48]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - SQL Table Pool: Completed — 11 checks  
[2026-09-16 12:36:48]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - Thread 'sql_table' completed  
[2026-09-16 12:36:50]  - INFO  - d4feeb03-eaa8-4069-b798-0ae89ec2b394 - api: Executed 13 checks across 1 table(s) for source='source_name'.
[16/Sep/2026 12:36:50] api
Calculating Metrics: 100%|████████████████████████████████████████████████████████████████| 38/38 [00:21<00:00,  1.75it/s]
[2026-09-16 12:37:02]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - GX Thread: Completed — 12 expectations evaluated  
[2026-09-16 12:37:02]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - Thread 'gx' completed
[2026-09-16 12:37:06]  - INFO  - a383802a-405c-494d-a9d2-0fb34041fdeb - api: Executed 23 checks across 1 table(s) for source="source_name".
[16/Sep/2026 12:37:06] "POST /check-run/ HTTP/1.1" 200 2224

Great Expectations version

1.15.1(also tried with latest version 1.23.0)

Python version

3.13.12

Execution engine / data source

Snowflake

If "Other", name the execution engine or data source

No response

Minimal reproduction

def _ephemeral_context(self, connect_args: Optional[dict] = None): gx_context = gx.get_context(mode="ephemeral") datasource_name = self._uid()

    try:
        # GX fluent API requires a connection_string (URL string), not an
        # execution_engine dict.  Extract the URL from the existing SA engine.
        conn_str = self._sa_engine.url.render_as_string(hide_password=False)

        # GX internally calls: sa.create_engine(connection_string, **kwargs)
        # So private_key (and any other connect_args) MUST be nested under
        # the "connect_args" key — passing them flat causes SQLAlchemy to
        # reject them as unknown create_engine() arguments.
        ca = connect_args or self._connect_args or {}
        if ca:
            gx_context.data_sources.add_sql(
                name=datasource_name,
                connection_string=conn_str,
                kwargs={"connect_args": ca},
            )
        else:
            gx_context.data_sources.add_sql(
                name=datasource_name,
                connection_string=conn_str,
            )

        logger.info("GX datasource registered successfully")

    except Exception as e:
        logger.error(f"GX datasource creation failed: {e}", exc_info=True)
        raise

    return gx_context, datasource_name

Additional environment details

  • Windows 11
  • Great Expectations: 1.15.1
  • Python: 3.13.12
  • Database: Snowflake
  • GX context: Ephemeral Data Context
  • Execution model: Python threads / ThreadPoolExecutor
  • Multiple validation checks can run concurrently.
  • Multiple API requests can also execute concurrently.

Additional context

No response

Source: fivetran/great_expectations

View original on GitHubView discussion on GitHub