[BUG] Custom eval_metric compatibility issues
Bug Report Checklist
- I provided code that demonstrates a minimal reproducible example.
- I confirmed bug exists on the latest mainline of AutoGluon via source install.
- I confirmed bug exists on the latest stable version of AutoGluon.
Describe the bug
I have given a custom eval_metric to the predictor, both by creating a custom metric with make_scorer and by providing the predictor with an uninstantiated root_mean_squared_log_error, but neither of these worked. The predictor seems to expect the scoring function to have a needs_proba attribute, which the latest version of sklearn does not have. This attribute has been deprecated since version 1.4 and is replaced by response_method. As far as I understand, version 1.2 should support version 1.5.2 of sklearn, so I see this as a bug.
Expected behavior I expect no errors.
To Reproduce
Dataset: https://www.kaggle.com/competitions/playground-series-s4e12
Code:
from sklearn.metrics import root_mean_squared_log_error
from autogluon.tabular import TabularPredictor
import pandas as pd
train = pd.read_csv('/kaggle/input/playground-series-s4e12/train.csv', index_col='id')
predictor = TabularPredictor(
problem_type='regression',
eval_metric=root_mean_squared_log_error,
label="Premium Amount",
verbosity=2
)
predictor.fit(
train_data=train,
time_limit=3600,
presets='best_quality'
)Screenshots / Logs
Installed Versions
INSTALLED VERSIONS
------------------
date : 2024-12-01
time : 12:35:55.222350
python : 3.10.14.final.0
OS : Linux
OS-release : 6.6.56+
Version : #1 SMP PREEMPT_DYNAMIC Sun Nov 10 10:07:59 UTC 2024
machine : x86_64
processor : x86_64
num_cores : 4
cpu_ram_mb : 32102.921875
cuda version : None
num_gpus : 0
gpu_ram_mb : []
avail_disk_size_mb : 19970
autogluon : None
autogluon.common : 1.2
autogluon.core : 1.2
autogluon.features : 1.2
autogluon.tabular : 1.2
boto3 : 1.26.100
catboost : 1.2.7
einops : None
fastai : 2.7.17
huggingface-hub : 0.25.1
hyperopt : 0.2.7
imodels : None
lightgbm : 4.2.0
matplotlib : 3.7.5
networkx : 3.3
numpy : 1.26.4
onnx : 1.17.0
onnxruntime : None
onnxruntime-gpu : None
pandas : 2.2.3
psutil : 5.9.3
pyarrow : 17.0.0
ray : 2.10.0
requests : 2.32.3
scikit-learn : 1.5.2
scikit-learn-intelex: 2024.7.0
scipy : 1.14.1
skl2onnx : None
spacy : 3.8.2
tabpfn : None
torch : 2.4.0+cpu
tqdm : 4.66.4
vowpalwabbit : None
xgboost : 2.0.3Source: autogluon/autogluon