[BUG] eval_results_table drops Series predictions with a non-default input index
[!WARNING] Before submitting a PR, please make sure that:
- A maintainer has triaged this issue and applied the
readylabel- This issue has no assignee
- No duplicate PR exists
PRs not meeting these requirements may be automatically closed.
Issues Policy acknowledgement
- I have read and agree to submit bug reports in accordance with the issues policy
Where did you encounter this bug?
Local machine
MLflow version
- Client:
3.16.1.dev0, currentmasteratf6252a80548ac13962788ac0cbd73799e8d51366 - Tracking server: local SQLite tracking URI
System information
- OS: Windows 11 (
10.0.26200) - Python:
3.12.5
Describe the problem
mlflow.models.evaluate can compute row-level metric values correctly but log None predictions in eval_results_table when evaluation data has a non-default pandas index and a callable model returns a Series with the normal RangeIndex.
The table assignment aligns the Series by label, while the metric path consumes predictions by position. Partially overlapping or reversed labels can therefore attach predictions to the wrong rows instead of making them null.
Expected: prediction values are attached to input rows by position, consistent with metric computation.
Actual: the logged table can disagree with the metrics.
Tracking information
Not applicable beyond the self-contained local SQLite setup below.
Code to reproduce issue
import tempfile
from pathlib import Path
import pandas as pd
import mlflow
from mlflow.metrics import MetricValue, make_metric
def predict(frame):
return pd.Series([101.0, 202.0])
def row_values(predictions, targets):
return MetricValue(scores=predictions.tolist())
work_dir = Path(tempfile.mkdtemp(prefix="mlflow-index-repro-"))
mlflow.set_tracking_uri(f"sqlite:///{work_dir / 'mlflow.db'}")
data = pd.DataFrame(
{"feature": [1, 2], "target": [101.0, 202.0]},
index=[10, 20],
)
with mlflow.start_run():
result = mlflow.models.evaluate(
predict,
data,
targets="target",
model_type="regressor",
evaluators="default",
evaluator_config={"log_model_explainability": False},
extra_metrics=[
make_metric(
eval_fn=row_values,
greater_is_better=True,
name="row_values",
)
],
)
table = pd.DataFrame(**result.artifacts["eval_results_table"].content)
print(table[["feature", "outputs"]].to_string(index=False))
assert table["outputs"].tolist() == [101.0, 202.0]Stack trace
feature outputs
1 None
2 None
Traceback (most recent call last):
...
AssertionErrorOther info / logs
A local two-file patch and native regression coverage are ready if maintainers agree with this direction.
Willingness to contribute
Yes. I would be willing to contribute a fix with guidance from the MLflow community.
What component(s) does this bug affect?
-
area/tracking -
area/model-registry -
area/scoring -
area/evaluation -
area/prompt -
area/tracing -
area/gateway -
area/projects -
area/uiux -
area/docs
Source: mlflow/mlflow