using `class_weights` with grid_search seems to provide incorrect results

catboost version: 1.2.0, Python package Operating System: Windows 10

The class_weights parameter provides incorrect results with grid_search for CatBoostClassifier. In the case below, best estimator will always have the FIRST class_weights value specified in the params list.

params = {'class_weights': [[1, 1], [1, 2]]} 
cb = CatBoostClassifier(eval_metric='F1')
cb.grid_search(params, train)  # train is a pool object

Then cb.get_all_params() gives:

...
'class_weights': [1, 1]
...

However, if we use params = {'class_weights': [[1, 2], [1, 1]]} to perform the grid search, then cb.get_all_params() will return:

...
'class_weights': [1, 2]
...

The 'live' output of grid search also has only 3 folds for the first parameter, then it doesn't show any verbose stuff at all. Nor does the plot work.

I believe the root cause of all these issues lies within how the class_weights parameter is implemented in CatBoost. When we use it, the calculated metric is weighted (more on this here and here). That means, if we indicate class_weights = [1, 1000], we'll get very good metrics. And grid search can't actually apply these metrics to compare models predictions, because it will always be biased towards the more imbalanced class_weights parameter. However, right now the grid search doesn't even compare the metrics for different values of class_weights, it just thinks that the first value in line is the best.

There are also a few open issues which seem to be connected with more or less the same problem: 1, 2, 3, 4(?)

Some notes and questions and thoughts:

1. Is there any reason that CatBoost uses the weighted metric by default? From my perspective this doesn't seem to have any point and causes a lot of confusion (see all the mentioned issues).

2. If I don't use grid_search, I can just specify:

   python复制

   cb = CatBoostClassifier(eval_metric='F1:use_weights=false')

   This will eventually give me the unweighted metric that I want. I thought I could fix the grid_search behaviour using the same approach. However, if I do this:

   python复制

   params = {'class_weights': [[1, 1], [1, 2]]} 
   cb = CatBoostClassifier(eval_metric='F1:use_weights=false')
   cb.grid_search(params, train)

   I'll get the following error instead:

   python复制

   CatBoostError: C:/Go_Agent/pipelines/BuildMaster/catboost.git/catboost/libs/metrics/metric.cpp:6376: If non-default weights for 
   objects are not set, the 'use_weights' parameter must not be specified.

3. All of the above is also true for the scale_pos_weight parameter.

4. I have tried only the 'F1' metric. But it seems, according to the open issues, this is not the only metric affected.

5. My first and naive suggestion is to make the metric calculated by catboost unweighted by default. This may probably help to solve some of the issues and make the interface more user-friendly.

Thank you!