Spark unexpected_index_query 无效 Python: F.expr(...) 条件未加引号
作者: vasilicommify创建于 2025年1月7日更新于 2026年9月17日
标签bughelp wantedgood first issueready-for-workclaimed🔔 reminder-sent
描述错误 "COMPLETE" result_format 输出 unexpected_index_query
unexpected_index_query 关键字的值无法用于提取有问题的行,即使是 expr() 内部的部分
原因之一是字符串值未被引号包围
value_set中有字符串,但在unexpected_index_query中这些字符串未被引号包围
{
"success": true,
"expectation_config": {
"type": "expect_column_values_to_be_in_set",
"kwargs": {
"batch_id": "my_spark_datasource-my_spark_dataframe",
"column": "in_set_str",
"value_set": [
"Val1",
"Val2"
]
},
"meta": {
"note": "column_values - without distinct"
}
},
"result": {
"element_count": 100,
"unexpected_count": 0,
"unexpected_percent": 0.0,
"partial_unexpected_list": [],
"missing_count": 11,
"missing_percent": 11.0,
"unexpected_percent_total": 0.0,
"unexpected_percent_nonmissing": 0.0,
"partial_unexpected_counts": [],
"unexpected_list": [],
"unexpected_index_query": "df.filter(F.expr((in_set_str IS NOT NULL) AND (NOT (in_set_str IN (Val1, Val2)))))"
},
"meta": {},
"exception_info": {
"raised_exception": false,
"exception_traceback": null,
"exception_message": null
}
},
2) 正则表达式未被引号包围
"unexpected_index_query": "df.filter(F.expr((email IS NOT NULL) AND (NOT RLIKE(email, ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$))))"
**重现步骤**
# expectation_suite = 具有字符串字段验证的任何期望套件
validation_definition = gx.ValidationDefinition(
data=batch_definition, suite=expectation_suite, name=validation_definition_name
)
validation_results = validation_definition.run(
batch_parameters={"dataframe": data_frame_to_check},
result_format = "COMPLETE")
validation_results
**预期行为** 在 `unexpected_index_query` 中,字符串和正则表达式应使用单引号或双引号包围,对于 Spark 而言
**环境(请填写以下信息):**
- Great Expectations 版本: 1.3.0
- 数据源: Spark
- 云环境: Databricks内容来源: fivetran/great_expectations