Spark unexpected_index_query is not valid Python: the F.expr(...) condition is emitted without quotes

Author: vasilicommifyCreated Jan 7, 2025Updated Sep 17, 2026

Labelsbughelp wantedgood first issueready-for-workclaimed🔔 reminder-sent

Describe the bug "COMPLETE" result_format outputs unexpected_index_query The value of the unexpected_index_query key cannot be used to extract rows with issue, even the part inside expr( ) One of the reasons is that string values are not quoted

- there are strings in value_set, but in unexpected_index_query these strings are not quoted

    {
      "success": true,
      "expectation_config": {
        "type": "expect_column_values_to_be_in_set",
        "kwargs": {
          "batch_id": "my_spark_datasource-my_spark_dataframe",
          "column": "in_set_str",
          "value_set": [
            "Val1",
            "Val2"
          ]
        },
        "meta": {
          "note": "column_values - without distinct"
        }
      },
      "result": {
        "element_count": 100,
        "unexpected_count": 0,
        "unexpected_percent": 0.0,
        "partial_unexpected_list": [],
        "missing_count": 11,
        "missing_percent": 11.0,
        "unexpected_percent_total": 0.0,
        "unexpected_percent_nonmissing": 0.0,
        "partial_unexpected_counts": [],
        "unexpected_list": [],
        "unexpected_index_query": "df.filter(F.expr((in_set_str IS NOT NULL) AND (NOT (in_set_str IN (Val1, Val2)))))"
      },
      "meta": {},
      "exception_info": {
        "raised_exception": false,
        "exception_traceback": null,
        "exception_message": null
      }
    },

regex expression is not quoted

"unexpected_index_query": "df.filter(F.expr((email IS NOT NULL) AND (NOT RLIKE(email, ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$))))"

To Reproduce

# expectation_suite = any expectation suite with with string fields validations

validation_definition = gx.ValidationDefinition(
    data=batch_definition, suite=expectation_suite, name=validation_definition_name
  )

validation_results = validation_definition.run(
                                      batch_parameters={"dataframe": data_frame_to_check}, 
                                      result_format = "COMPLETE")            
validation_results

Expected behavior string and regex expressions should be quoted in unexpected_index_query - either with single or double quates for spark

Environment (please complete the following information):

Great Expectations Version: 1.3.0
Data Source: Spark
Cloud environment: Databricks

Source: fivetran/great_expectations

View original on GitHub View discussion on GitHub