Use fixed parameter information for acquisition function evaluations when using`PartialFixedSampler`

Motivation

import optuna


def objective(trial: optuna.Trial) -> float:
    x = trial.suggest_float("x", -5, 5)
    y = trial.suggest_float("y", -5, 5)
    return x**2 + y**2

base_sampler = optuna.samplers.GPSampler()
# or base_sampler = optuna.samplers.TPESampler(multivariate=True)
study = optuna.create_study(sampler=base_sampler)
study.optimize(objective, n_trials=20)
study.sampler = optuna.samplers.PartialFixedSampler(fixed_params={"x": study.best_params["x"]}, base_sampler=base_sampler)
study.optimize(objective, n_trials=10)

In the example above, the surrogate fitting should be performed along with the fixed parameters as well, but the fixed parameters are simply dropped during the training and the acquisition function optimization. Since the fixed parameters also affect output prediction (and of course the acquisition function landscape as well), they should also be accounted.

Description

1. Use the fixed parameters for surrogate fitting
2. Optimize unfixed parameters while fixing the fixed paraemters during the acquisition function optimization

This is very useful especially when we use Optuna for very expensive HPO, where we would like to tune, for example, learning rate first and tune the rest thereafter.

Alternatives (optional)

No response

Additional context (optional)

No response