[BUG] Spark: a dotted column name plus unexpected_index_column_names returns an empty result dict and a false failure

What's wrong

On Spark, validating any expectation whose domain column name contains a dot (a flat column literally named Data.Entrega, not a struct) while also passing unexpected_index_column_names raises InvalidMetricAccessorDomainKwargsKeyError: The unexpected_index_column 'Data.Entrega' does not exist in Spark DataFrame. The error is swallowed into exception_info, and the expectation's result comes back as {} — no element_count, no unexpected_count, no unexpected_index_list — with success: False.

Dotted column names were fixed on Spark in 1.17.1 (#11851), and that fix works: the same column validates fine without index columns. What #11851 did not reach is the index path. Its normalizer hands _spark_map_condition_index the backticked name, and that function (great_expectations/expectations/metrics/map_metric_provider/map_condition_auxilliary_methods.py:778-782) checks `Data.Entrega` verbatim against filtered.columns, which holds Data.Entrega.

Impact

False failure: with data that has no violations at all, the same configuration still reports success: False, because the metric raised before any count was computed. Spark only, 1.17.1 through 1.23.0. Triggered by the combination of a dotted domain column and any unexpected_index_column_names (the index column itself can be dot-free); COMPLETE and SUMMARY fail identically. Pandas returns a complete result for the same input.

Workaround: drop unexpected_index_column_names for dotted columns and lose the index list.

Reproduction

Reproduced on develop at 20e5722d9 (1.23.0 line), Python 3.11, pyspark 3.5.8.

Add to tests/integration/data_sources_and_expectations/data_sources/test_spark_column_names_with_dots.py (the #11851 regression file, which never sets unexpected_index_column_names and so never enters this path):

ID_COLUMN = "ID"

DATA_WITH_A_NULL = pd.DataFrame(
    {
        ID_COLUMN: [1, 2],
        COLUMN_WITH_DOT: ["2024-01-01", None],
    }
)


@parameterize_batch_for_data_sources(
    data_source_configs=[SparkFilesystemCsvDatasourceTestConfig()],
    data=DATA_WITH_A_NULL,
)
def test_spark_column_with_dot_in_name_returns_unexpected_index_list(
    batch_for_datasource: Batch,
) -> None:
    """A dotted domain column must not empty the result when index columns are requested."""
    result = batch_for_datasource.validate(
        gxe.ExpectColumnValuesToNotBeNull(column=COLUMN_WITH_DOT),
        result_format={
            "result_format": "COMPLETE",
            "unexpected_index_column_names": [ID_COLUMN],
        },
    )
    assert result.result, result.exception_info  # empty when a metric raised
    assert result.result["unexpected_count"] == 1
    assert result.result["unexpected_index_list"] == [{ID_COLUMN: 2, COLUMN_WITH_DOT: None}]

pytest tests/integration/data_sources_and_expectations/data_sources/test_spark_column_names_with_dots.py::test_spark_column_with_dot_in_name_returns_unexpected_index_list -m spark

Observed:

E   AssertionError: {'MetricConfigurationID(metric_name=\'column_values.nonnull.unexpected_index_list\', ...)': {
E     ... map_condition_auxilliary_methods.py", line 780, in _spark_map_condition_index
E         raise gx_exceptions.InvalidMetricAccessorDomainKwargsKeyError(
E     'exception_message': "Error: The unexpected_index_column '`Data.Entrega`' does not exist in Spark DataFrame. Please check your configuration and try again.", 'raised_exception': True}}
E   assert {}

Expected:

unexpected_count == 1
unexpected_index_list == [{"ID": 2, "Data.Entrega": None}]

Requirements

1. When a Spark expectation's domain column name contains a dot and unexpected_index_column_names is set, the result must carry the full result dict — counts, unexpected_index_list, and unexpected_index_query — exactly as it does for a dot-free column.
2. Keys in unexpected_index_list entries must be the plain column names (Data.Entrega, ID), never backticked.
3. unexpected_index_query for a dotted column must remain correctly backticked (it is today, via the same normalized name).
4. Dot-free columns, nested struct paths (Data.evt.id, covered by test_spark_nested_columns_unexpected_index.py), and the pandas/SQL index paths must be unchanged.
5. The InvalidMetricAccessorDomainKwargsKeyError for a genuinely missing index column must still be raised.

Out of scope: the content of the Spark unexpected_index_query string itself (#10827); column_list domains through the multicolumn condition builder, which have a related but separately triggered failure.

Context

• First place to look: map_condition_auxilliary_methods.py:735-739, where the domain column from accessor_domain_kwargs is appended to columns_to_keep. The name arrives backticked because metrics/util.py:891-904 (from #11851) returns the backticked form of a matched table.column_types name and _get_dbms_compatible_metric_domain_kwargs writes it into metric_domain_kwargs["column"].
• Unwrap at the append site rather than loosening the guard at :778-782: columns_to_keep is used twice more downstream, and both uses want the plain name — :787-800 re-quotes each entry (an already-backticked name would be double-quoted), and _get_spark_customized_unexpected_index_list uses the entries as the keys of the returned index dicts. Verified locally: unwrapping the backticks at the append site turns the test above green with exactly the expected unexpected_index_list.
• Two fixes shipped in 1.17.1 eight days apart and do not compose: #11835 taught the index path about dotted paths assuming an unbackticked name; #11851 then changed what arrives. The unexpected_index_column wording in the error is misleading here — the offending name came from the domain, not from unexpected_index_column_names.
• Ruled out: not a struct/nested issue (the column is flat, created via the harness CSV); not specific to which column is named as the index; not one expectation's bug (ExpectColumnPairValuesToBeEqual fails the same way); not a result_format verbosity effect.