`SchemaValidationError` relevance heuristic is flawed

What happened?

[!NOTE] Originally wrote as comment in #2913, but this will require more work to fix

While debugging some tests for (#3750), I've begun to think the underlying assumption here is flawed:

...validated against multiple schemas and its parent is a common anyOf validator. The error messages produced from these cases are usually very similar and we just take the shortest one.

https://github.com/vega/altair/blob/be5e9ecd61d099c847dca44fb3a4283940c1e3b8/tools/schemapi/schemapi.py#L452-L479

This assumption conflicts directly with the channel wrappers, since the "anyOf" is how they are differentiated.

https://github.com/vega/altair/blob/be5e9ecd61d099c847dca44fb3a4283940c1e3b8/tools/generate_schema_wrapper.py#L816

Examples

These are all tested against (https://github.com/vega/altair/pull/3750/commits/8fb2661555ff9f70db381b27506fdb3eb31b5a6d)

import altair as alt

chart = alt.Chart().mark_point()

one_err = alt.value(1, bin=True)
two_err = alt.value(1, bin=True, axis=1)
two_err_2 = alt.value(1, bin=True, aggregate="sum")
three_err = alt.value(1, bin=True, axis=1, aggregate="sum")

We're trying to match alt.YValue. The most helpful thing would be identifying all of the invalid properties.

Naively picking the shortest message is an unfit heuristic. This can be observed by seeing how the final message changes in an unpredicatble way, simply by adding more invalid properties:

>>> chart.encode(y=one_err)
SchemaValidationError: `YValue` has no parameter named 'bin'
Existing parameter names are:
value

>>> chart.encode(y=two_err)
SchemaValidationError: '1' is an invalid value for `axis`. Valid values are of type `Mapping[str, Any] | None`.

>>> chart.encode(y=two_err_2)
SchemaValidationError: `YValue` has no parameter named 'value'
Existing parameter names are:
value

>>> chart.encode(y=three_err)
SchemaValidationError: '1' is an invalid value for `axis`. Valid values are of type `Mapping[str, Any] | None`.

Exploring the error tree

Diving into the error tree for two_err_2 gives us some more clarity:

Capturing an error

import altair as alt
from altair.utils.schemapi import SchemaValidationError

two_err_2 = alt.value(1, bin=True, aggregate="sum")
try:
    alt.Chart().mark_point().encode(y=two_err_2).to_dict()
except SchemaValidationError as err:
    some = err

Spec being validated

>>> some.instance
{'value': 1, 'bin': True, 'aggregate': 'sum'}

altair transformed message

Since (https://github.com/vega/altair/pull/3750/commits/ea647ebc83e84c0cc0b9731b3acf99b35c4393a4) we now at least get YValue included:

>>> some.message
"`YValue` has no parameter named 'value'\n\nExisting parameter names are:\nvalue   \n\nSee the help for `YValue` to read the full description of these parameters"

The first error

Backtracking to what the message originally looked like from jsonschema.

I think it is pretty clear this is not the error we want:

>>> some._original_message 
"Additional properties are not allowed ('value' was unexpected)"

Parent message is more useful

Going back one level (.parent) starts to make more sense:

>>> some.parent.message
"{'value': 1, 'bin': True, 'aggregate': 'sum'} is not valid under any of the given schemas"

Significant backtracking

It is still possible to undo all of the work from validate_jsonschema

https://github.com/vega/altair/blob/be5e9ecd61d099c847dca44fb3a4283940c1e3b8/tools/schemapi/schemapi.py#L135-L170

I wouldn't want to solve the problem this way, but an example of backtracking to a useful context:

tree_nav = {
    err.parent.schema["anyOf"][idx]["$ref"].removeprefix("#/definitions/"): err.message
    for idx, err in enumerate(some.parent.context)
}
>>> tree_nav
{'PositionFieldDef': "Additional properties are not allowed ('value' was unexpected)",
 'PositionDatumDef': "Additional properties are not allowed ('aggregate', 'bin', 'value' were unexpected)",
 'PositionValueDef': "Additional properties are not allowed ('aggregate', 'bin' were unexpected)"}

So the hacky way to do this would be replacing the error we got from 'PositionFieldDef' with the one for 'PositionValueDef'. For our {'value': 1, 'bin': True, 'aggregate': 'sum'}, we identify the two extra invalid properties "...('aggregate', 'bin' were unexpected)"

What would you like to happen instead?

I think #3750 offers an improvement, but really we need to fix this before raising in validate_jsonschema. By that stage, we've dropped errors that are better candidates which makes fixing it later much more difficult.

Maybe this would be something to pick up when revisiting #3547, as that PR contains a lot of refactoring/redocumenting in tools.schemapi.schemapi.py

Which version of Altair are you using?

5.6.0dev