Filter out incompatible hyperparameters
Motivation
In many machine learning workflows, especially those based on sklearn.Pipeline, hyperparameters are interdependent across steps. Some examples:
If PCA(n_components=n) is used, a following SelectKBest(k) must satisfy k ≤ n
If PCA(whiten=True), the svd_solver must be 'full'
If LogisticRegression(penalty='l1'), then solver must be 'liblinear' or 'saga'
If dimensionality reduction is applied, subsequent estimators or selectors must operate on compatible feature dimensions
Currently, the only way to handle these constraints is manually inside the objective function, returning float("nan") for invalid combinations. This results in a large number of failed trials, which is wasteful and makes grid or brute-force search particularly inefficient.
Description
Optuna should automatically filter out incompatible parameter combinations before trials are run, by introspecting:
- Pipeline structure (e.g., from
sklearn.pipeline.Pipeline) - Parameter relationships (e.g., dimensionality reductions, required parameter dependencies)
- Common model constraints (e.g., solver/penalty validity in scikit-learn models)
This filtering should occur before the optimization begins, especially in exhaustive samplers like GridSampler and BruteForceSampler, to:
- Avoid unnecessary trial runs
- Reduce memory usage, compute cost, and log clutter
- Make Optuna smarter when working with structured ML pipelines
This functionality would make Optuna pipeline-aware, particularly helpful for users working with modular, dynamic, or stacked models.
Alternatives (optional)
If full automatic detection is not feasible or too complex, Optuna could allow users to define constraint rules declaratively, e.g.:
study = optuna.create_study(
direction='maximize',
sampler=GridSampler(search_space),
constraints=[
('pca__n_components', '>=', 'selectkbest__k'),
('logisticregression__penalty=l1', 'logisticregression__solver', ['saga', 'liblinear']),
]
)These could be evaluated before the trial begins, allowing filtering without needing to enter the objective() or track failed trials.
Additional context (optional)
In my current workflow, I'm performing exhaustive hyperparameter search over a modular and dynamic scikit-learn pipeline, with steps including:
- Preprocessing:
RobustScaler,PowerTransformer - Resampling:
SMOTE,ADASYN,SMOTEENN - Dimensionality Reduction:
PCA - Feature Selection:
SelectKBest,RFE,SequentialFeatureSelector - Estimators:
LogisticRegression
My grid search space results in over 307,000 hyperparameter combinations. I use GridSampler to ensure complete coverage. However, due to step dependencies (e.g., PCA → feature selection) and model-specific constraints (e.g., incompatible solver/penalty pairs in LogisticRegression), a large proportion of the trials fail .
Despite careful objective design (returning float('nan') for invalid configs), these failed trials:
- Still consume compute resources and memory
- Still generate noise in logs and warnings
- Still slow down the overall process
This leads to massive inefficiency that could be entirely avoided if Optuna could pre-filter these invalid combinations by understanding structural incompatibilities in the pipeline.
I'm adding here my DynamicSearchObjective class that unfortunately is running all parameters combination in GridSample, even those incompatible ones:
import numpy as np
import optuna
from sklearn.model_selection import cross_val_score
from sklearn.utils import check_X_y
import torch
class DynamicSearchObjective:
def __init__(self, pipeline, steps, X_train, y_train, scoring='roc_auc'):
self.pipeline = pipeline
self.steps = steps
self.X_train, self.y_train = check_X_y(X_train, y_train, accept_sparse=True)
self.n_features = self.X_train.shape[1]
self.min_class_count = np.bincount(self.y_train)[np.bincount(self.y_train) > 0].min()
self.scoring = scoring
self.search_space = self._get_search_space()
self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
def _get_search_space(self):
search_space = {}
for step_name, step_obj in self.steps:
cname = step_obj.__class__.__name__.lower()
if cname == 'robustscaler':
search_space[f'{step_name}__quantile_range'] = ['10_90', '25_75']
elif cname == 'powertransformer':
search_space[f'{step_name}__standardize'] = [True, False]
elif cname == 'smote':
search_space[f'{step_name}__k_neighbors'] = list(range(1, min(self.min_class_count - 1, 5)))
search_space[f'{step_name}__sampling_strategy'] = ['auto', 'minority', 'not majority', 'all']
elif cname == 'adasyn':
search_space[f'{step_name}__n_neighbors'] = list(range(1, min(self.min_class_count - 1, 5)))
search_space[f'{step_name}__sampling_strategy'] = ['auto', 'minority', 'not majority', 'all']
elif cname == 'pca':
search_space[f'{step_name}__n_components'] = [3, 5, 9, 15, self.n_features]
search_space[f'{step_name}__whiten'] = [True, False]
search_space[f'{step_name}__svd_solver'] = ['auto', 'full', 'arpack', 'randomized']
elif cname == 'selectkbest':
search_space[f'{step_name}__k'] = [3, 5, 9, 15, self.n_features]
elif cname == 'sequentialfeatureselector':
search_space[f'{step_name}__direction'] = ['forward', 'backward']
elif cname == 'rfe':
search_space[f'{step_name}__n_features_to_select'] = [3, 5, 9, 15, self.n_features]
elif cname == 'logisticregression':
search_space[f'{step_name}__C'] = list(np.logspace(-3, 0, num=4))
search_space[f'{step_name}__penalty'] = ['l1', 'l2']
search_space[f'{step_name}__solver'] = ['liblinear', 'saga', 'lbfgs']
search_space[f'{step_name}__class_weight'] = [None, 'balanced']
elif cname == 'xgbclassifier':
search_space[f'{step_name}__n_estimators'] = list(range(100, 501, 100))
search_space[f'{step_name}__learning_rate'] = list(np.logspace(-3, 0, num=4))
search_space[f'{step_name}__max_depth'] = list(np.linspace(3, 10, num=4).astype(int))
search_space[f'{step_name}__subsample'] = list(np.linspace(0.5, 1.0, num=3))
search_space[f'{step_name}__colsample_bytree'] = list(np.linspace(0.5, 1.0, num=3))
elif cname == 'randomforestclassifier':
search_space[f'{step_name}__n_estimators'] = list(range(100, 501, 100))
search_space[f'{step_name}__max_depth'] = list(np.linspace(3, 10, num=4).astype(int))
search_space[f'{step_name}__min_samples_split'] = list(np.linspace(3, 10, num=4).astype(int))
search_space[f'{step_name}__min_samples_leaf'] = list(np.linspace(1, 5, num=3).astype(int))
elif cname == 'xgbregressor' or cname == 'randomforestregressor':
search_space[f'{step_name}__n_estimators'] = list(range(100, 501, 100))
elif cname == 'ridge':
search_space[f'{step_name}__alpha'] = list(np.logspace(-3, 2, num=10))
elif cname == 'lasso':
search_space[f'{step_name}__alpha'] = list(np.logspace(-4, 0, num=10))
elif cname == 'elasticnet':
search_space[f'{step_name}__alpha'] = list(np.logspace(-4, 0, num=10))
search_space[f'{step_name}__l1_ratio'] = list(np.linspace(0.1, 0.9, num=5))
return search_space
def _suggest_and_store(self, trial, param_name, labels):
label = trial.suggest_categorical(param_name, labels)
q1, q2 = label.split('_')
real_value = (float(q1), float(q2))
trial.set_user_attr(param_name, real_value)
return real_value
def __call__(self, trial: optuna.trial.Trial):
params = {}
current_n_features = self.n_features
for step_name, _ in self.steps:
for param_name, values in self.search_space.items():
if not param_name.startswith(f"{step_name}__"):
continue
if isinstance(values, list):
if isinstance(values[0], str) or isinstance(values[0], (str, type(None))):
if param_name in ['robustscaler__quantile_range']:
value = self._suggest_and_store(trial, param_name, values)
else:
value = trial.suggest_categorical(param_name, values)
elif isinstance(values[0], bool):
value = trial.suggest_categorical(param_name, values)
elif isinstance(values[0], int):
if step_name in ['smote', 'adasyn']:
value = trial.suggest_int(param_name, min(values), min(self.min_class_count - 1, max(values)))
elif param_name in [c+'__n_components' for c in ['pca', 'selectkbest', 'rfe']]:
value = trial.suggest_int(param_name, min(values), min(current_n_features, max(values)))
current_n_features = value
else:
value = trial.suggest_int(param_name, min(values), max(values))
elif isinstance(values[0], float):
value = trial.suggest_float(param_name, min(values), max(values))
else:
raise ValueError(f"Unsupported type in search space: {type(values[0])}")
else:
raise ValueError(f"Unsupported search space format for {param_name}")
params[param_name] = value
trial.set_user_attr(param_name, value)
if self.device == 'cuda' and step_name in ['xgbclassifier', 'xgbregressor']:
params[f'{step_name}__tree_method'] = 'hist'
params[f'{step_name}__device'] = 'cuda'
try:
self.pipeline.set_params(**params)
score = cross_val_score(self.pipeline, self.X_train, self.y_train, cv=5, scoring=self.scoring, error_score='ignore').mean()
return score
except Exception:
return float('nan')Source: optuna/optuna