Skip to content

Hyperparameter tuning

Optuna-based tuning and the gin helpers that turn hyperparameter bindings into a search space. See Hyperparameter tuning.

icu_benchmarks.tuning.hyperparameters

Functions:

Name Description
choose_and_bind_hyperparameters_scikit_optimize

Choose hyperparameters to tune and bind them to gin.

choose_and_bind_hyperparameters_optuna

Choose hyperparameters to tune and bind them to gin. Uses Optuna for hyperparameter optimization.

make_sampler

Instantiates the sampler class; the GP sampler explores n_initial_points random trials first.

suggest_hyperparameter

Samples one hyperparameter from its gin bound.

find_checkpoint

Find the latest checkpoint in the log directory.

choose_and_bind_hyperparameters_scikit_optimize

choose_and_bind_hyperparameters_scikit_optimize(do_tune: bool, data_dir: Path, log_dir: Path, seed: int, run_mode: RunMode = classification, checkpoint: str = None, scopes: list[str] = [], n_initial_points: int = 3, n_calls: int = 20, folds_to_tune_on: int = None, checkpoint_file: str = 'hyperparameter_tuning_logs.json', generate_cache: bool = False, load_cache: bool = False, debug: bool = False, verbose: bool = False, wandb: bool = False)

Choose hyperparameters to tune and bind them to gin.

Parameters:

Name Type Description Default
wandb bool

Whether we use wandb or not.

False
load_cache bool

Load cached data if available.

False
generate_cache bool

Generate cache data.

False
do_tune bool

Whether to tune hyperparameters or not.

required
data_dir Path

Path to the data directory.

required
log_dir Path

Path to the log directory.

required
seed int

Random seed.

required
run_mode RunMode

The run mode of the experiment.

classification
checkpoint str

Name of the checkpoint run to load previously explored hyperparameters from.

None
scopes list[str]

List of gin scopes to search for hyperparameters to tune.

[]
n_initial_points int

Number of initial points to explore.

3
n_calls int

Number of iterations to optimize the hyperparameters.

20
folds_to_tune_on int

Number of folds to tune on.

None
checkpoint_file str

Name of the checkpoint file.

'hyperparameter_tuning_logs.json'
debug bool

Whether to load less data.

False
verbose bool

Set to true to increase log output.

False

Raises:

Type Description
ValueError

If checkpoint is not None and the checkpoint does not exist.

Source code in icu_benchmarks/tuning/hyperparameters.py
@gin.configurable("tune_hyperparameters_deprecated")
def choose_and_bind_hyperparameters_scikit_optimize(
    do_tune: bool,
    data_dir: Path,
    log_dir: Path,
    seed: int,
    run_mode: RunMode = RunMode.classification,
    checkpoint: str = None,
    scopes: list[str] = [],
    n_initial_points: int = 3,
    n_calls: int = 20,
    folds_to_tune_on: int = None,
    checkpoint_file: str = "hyperparameter_tuning_logs.json",
    generate_cache: bool = False,
    load_cache: bool = False,
    debug: bool = False,
    verbose: bool = False,
    wandb: bool = False,
):
    """Choose hyperparameters to tune and bind them to gin.

    Args:
        wandb: Whether we use wandb or not.
        load_cache: Load cached data if available.
        generate_cache: Generate cache data.
        do_tune: Whether to tune hyperparameters or not.
        data_dir: Path to the data directory.
        log_dir: Path to the log directory.
        seed: Random seed.
        run_mode: The run mode of the experiment.
        checkpoint: Name of the checkpoint run to load previously explored hyperparameters from.
        scopes: List of gin scopes to search for hyperparameters to tune.
        n_initial_points: Number of initial points to explore.
        n_calls: Number of iterations to optimize the hyperparameters.
        folds_to_tune_on: Number of folds to tune on.
        checkpoint_file: Name of the checkpoint file.
        debug: Whether to load less data.
        verbose: Set to true to increase log output.

    Raises:
        ValueError: If checkpoint is not None and the checkpoint does not exist.
    """
    logging.warning(
        "This function is deprecated and will be removed in the future. Use choose_and_bind_hyperparameters_optuna instead."
    )
    hyperparams = {}

    if len(scopes) == 0 or folds_to_tune_on is None:
        logging.warning("No scopes and/or folds to tune on, skipping tuning.")
        return

    # Collect hyperparameters.
    hyperparams_bounds, hyperparams_names = collect_bound_hyperparameters(hyperparams, scopes)

    if do_tune and not hyperparams_bounds:
        logging.info("No hyperparameters to tune, skipping tuning.")
        return

    # Attempt checkpoint loading
    configuration, evaluation = None, None
    if checkpoint:
        checkpoint_path = checkpoint / checkpoint_file
        if not checkpoint_path.exists():
            logging.warning(f"Hyperparameter checkpoint {checkpoint_path} does not exist.")
            logging.info("Attempting to find latest checkpoint file.")
            checkpoint_path = find_checkpoint(log_dir.parent, checkpoint_file)
        # Check if we found a checkpoint file
        if checkpoint_path:
            n_calls, configuration, evaluation = load_checkpoint(checkpoint_path, n_calls)
            # Check if we surpassed maximum tuning iterations
            if n_calls <= 0:
                logging.log(
                    TUNE,
                    "No more hyperparameter tuning iterations left, skipping tuning.",
                )
                logging.info("Training with these hyperparameters:")
                bind_gin_params(hyperparams_names, configuration[np.argmin(evaluation)])  # bind best hyperparameters
                return
        else:
            logging.warning("No checkpoint file found, starting from scratch.")

    # Function that trains the model with the given hyperparameters.
    def bind_params_and_train(hyperparams):
        with tempfile.TemporaryDirectory(dir=log_dir) as temp_dir:
            bind_gin_params(hyperparams_names, hyperparams)
            if not do_tune:
                return 0
            return execute_repeated_cv(
                data_dir,
                Path(temp_dir),
                seed,
                mode=run_mode,
                cv_repetitions_to_train=1,
                cv_folds_to_train=folds_to_tune_on,
                generate_cache=generate_cache,
                load_cache=load_cache,
                test_on="val",
                debug=debug,
                verbose=verbose,
                wandb=wandb,
                # SHAP values are expensive and unused while tuning
                explain_features=False,
            )

    header = ["ITERATION"] + hyperparams_names + ["LOSS AT ITERATION"]

    def tune_step_callback(res):
        with open(log_dir / checkpoint_file, "w") as f:
            data = {
                "x_iters": res.x_iters,
                "func_vals": res.func_vals,
            }
            f.write(json.dumps(data, cls=JsonResultLoggingEncoder))
            table_cells = [len(res.x_iters)] + res.x_iters[-1] + [res.func_vals[-1]]
            highlight = res.x_iters[-1] == res.x  # highlight if best so far
            log_table_row(header, TUNE)
            log_table_row(table_cells, TUNE, align=Align.RIGHT, header=header, highlight=highlight)
            wandb_log({"hp-iteration": len(res.x_iters)})

    if do_tune:
        log_full_line("STARTING TUNING", level=TUNE, char="=")
        logging.log(
            TUNE,
            f"Applying Bayesian Optimization from {n_initial_points} points in {n_calls} "
            f"iterations on {folds_to_tune_on} folds.",
        )
        log_table_row(header, TUNE)
    else:
        logging.log(TUNE, "Hyperparameter tuning disabled")
        if configuration:
            # We have loaded a checkpoint, use the best hyperparameters.
            logging.info("Training with the best hyperparameters from loaded checkpoint:")
            bind_gin_params(hyperparams_names, configuration[np.argmin(evaluation)])
        else:
            logging.log(TUNE, "Choosing hyperparameters randomly from bounds.")
            n_initial_points = 1
            n_calls = 1

    # Call gaussian process. To choose a random set of hyperparameters this functions is also called.
    res = gp_minimize(
        bind_params_and_train,
        hyperparams_bounds,
        x0=configuration,
        y0=evaluation,
        n_calls=n_calls,
        n_initial_points=n_initial_points,
        random_state=seed,
        noise=1e-10,  # The models are deterministic, but noise is needed for the gp to work.
        callback=tune_step_callback if do_tune else None,
    )
    logging.disable(level=NOTSET)

    if do_tune:
        log_full_line("FINISHED TUNING", level=TUNE, char="=", num_newlines=4)

    logging.info("Training with these hyperparameters:")
    bind_gin_params(hyperparams_names, res.x)

choose_and_bind_hyperparameters_optuna

choose_and_bind_hyperparameters_optuna(do_tune: bool, data_dir: Path, log_dir: Path, seed: int, run_mode: RunMode = classification, checkpoint: str = None, scopes: list[str] = [], n_initial_points: int = 3, n_calls: int = 20, sampler=GPSampler, folds_to_tune_on: int = None, checkpoint_file: str = 'hyperparameter_tuning_logs.db', generate_cache: bool = False, load_cache: bool = False, debug: bool = False, verbose: bool = False, wandb: bool = False, plot: bool = True)

Choose hyperparameters to tune and bind them to gin. Uses Optuna for hyperparameter optimization.

Parameters:

Name Type Description Default
plot bool

Whether to plot hyperparameter importances.

True
sampler

The sampler to use for hyperparameter optimization.

GPSampler
wandb bool

Whether we use wandb or not.

False
load_cache bool

Load cached data if available.

False
generate_cache bool

Generate cache data.

False
do_tune bool

Whether to tune hyperparameters or not.

required
data_dir Path

Path to the data directory.

required
log_dir Path

Path to the log directory.

required
seed int

Random seed.

required
run_mode RunMode

The run mode of the experiment.

classification
checkpoint str

Name of the checkpoint run to load previously explored hyperparameters from.

None
scopes list[str]

List of gin scopes to search for hyperparameters to tune.

[]
n_initial_points int

Number of initial points to explore.

3
n_calls int

Number of iterations to optimize the hyperparameters.

20
folds_to_tune_on int

Number of folds to tune on.

None
checkpoint_file str

Name of the checkpoint file.

'hyperparameter_tuning_logs.db'
debug bool

Whether to load less data.

False
verbose bool

Set to true to increase log output.

False

Raises:

Type Description
ValueError

If tuning is disabled and the checkpoint does not exist. With tuning, a new study is started instead.

Source code in icu_benchmarks/tuning/hyperparameters.py
@gin.configurable("tune_hyperparameters")
def choose_and_bind_hyperparameters_optuna(
    do_tune: bool,
    data_dir: Path,
    log_dir: Path,
    seed: int,
    run_mode: RunMode = RunMode.classification,
    checkpoint: str = None,
    scopes: list[str] = [],
    n_initial_points: int = 3,
    n_calls: int = 20,
    sampler=optuna.samplers.GPSampler,
    folds_to_tune_on: int = None,
    checkpoint_file: str = "hyperparameter_tuning_logs.db",
    generate_cache: bool = False,
    load_cache: bool = False,
    debug: bool = False,
    verbose: bool = False,
    wandb: bool = False,
    plot: bool = True,
):
    """Choose hyperparameters to tune and bind them to gin. Uses Optuna for hyperparameter optimization.

    Args:
        plot: Whether to plot hyperparameter importances.
        sampler: The sampler to use for hyperparameter optimization.
        wandb: Whether we use wandb or not.
        load_cache: Load cached data if available.
        generate_cache: Generate cache data.
        do_tune: Whether to tune hyperparameters or not.
        data_dir: Path to the data directory.
        log_dir: Path to the log directory.
        seed: Random seed.
        run_mode: The run mode of the experiment.
        checkpoint: Name of the checkpoint run to load previously explored hyperparameters from.
        scopes: List of gin scopes to search for hyperparameters to tune.
        n_initial_points: Number of initial points to explore.
        n_calls: Number of iterations to optimize the hyperparameters.
        folds_to_tune_on: Number of folds to tune on.
        checkpoint_file: Name of the checkpoint file.
        debug: Whether to load less data.
        verbose: Set to true to increase log output.

    Raises:
        ValueError: If tuning is disabled and the checkpoint does not exist. With tuning, a new study is started instead.
    """
    hyperparams = {}

    # Checked first so that a run that skips tuning does not silently ignore the checkpoint.
    # Loading a missing sqlite storage would create an empty file and fail on the missing study
    if not do_tune and checkpoint and not Path(checkpoint).exists():
        raise ValueError(f"Hyperparameter checkpoint {checkpoint} does not exist.")

    if len(scopes) == 0 or folds_to_tune_on is None:
        logging.warning("No scopes and/or folds to tune on, skipping tuning.")
        return

    # Collect hyperparameters.
    hyperparams_bounds, hyperparams_names = collect_bound_hyperparameters(hyperparams, scopes)

    if do_tune and not hyperparams_bounds:
        logging.info("No hyperparameters to tune, skipping tuning.")
        return

    # Function that trains the model with the given hyperparameters.

    header = ["ITERATION"] + hyperparams_names + ["LOSS AT ITERATION"]

    # Optuna objective function
    def objective(trial, hyperparams_bounds, hyperparams_names):
        # Optuna objective function
        hyperparams = {}
        logging.info(f"Bounds: {hyperparams_bounds}, Names: {hyperparams_names}")
        for name, value in zip(hyperparams_names, hyperparams_bounds):
            hyperparams[name] = suggest_hyperparameter(trial, name, value)
        return bind_params_and_train(hyperparams)

    def tune_step_callback(study: optuna.study.Study, trial: optuna.trial.FrozenTrial):
        table_cells = [
            str(len(study.trials)),
            *list(study.trials[-1].params.values()),
            study.trials[-1].value,
        ]
        highlight = study.trials[-1] == study.best_trial  # highlight if best so far
        log_table_row(header, TUNE)
        log_table_row(table_cells, TUNE, align=Align.RIGHT, header=header, highlight=highlight)
        wandb_log({"HP-optimization-iteration": len(study.trials)})

    if do_tune:
        log_full_line("STARTING TUNING", level=TUNE, char="=")
        logging.log(
            TUNE,
            f"Applying {sampler} from {n_initial_points} points in {n_calls} iterations on {folds_to_tune_on} folds.",
        )
        log_table_row(header, TUNE)
    else:
        logging.log(TUNE, "Hyperparameter tuning disabled")
        if checkpoint:
            study = optuna.load_study(study_name="tuning", storage="sqlite:///" + str(checkpoint))
            configuration = study.best_params
            # We have loaded a checkpoint, use the best hyperparameters.
            logging.info("Training with the best hyperparameters from loaded checkpoint:")
            bind_gin_params(configuration)
            return
        else:
            logging.log(
                TUNE,
                "Choosing hyperparameters randomly from bounds using hp tuning as no earlier checkpoint supplied.",
            )
            n_initial_points = 1
            n_calls = 1

    def bind_params_and_train(hyperparams):
        with tempfile.TemporaryDirectory(dir=log_dir) as temp_dir:
            bind_gin_params(hyperparams)
            if not do_tune:
                return 0
            score = execute_repeated_cv(
                data_dir,
                Path(temp_dir),
                seed,
                mode=run_mode,
                cv_repetitions_to_train=1,
                cv_folds_to_train=folds_to_tune_on,
                generate_cache=generate_cache,
                load_cache=load_cache,
                test_on="val",
                debug=debug,
                verbose=verbose,
                wandb=wandb,
                # SHAP values are expensive and unused while tuning
                explain_features=False,
            )
            logging.info(f"Score: {score}")
            return score

    sampler = make_sampler(sampler, seed, n_initial_points)
    pruner = optuna.pruners.HyperbandPruner()
    # Optuna study
    # Attempt checkpoint loading
    if checkpoint and checkpoint.exists():
        # logging.info("Attempting to find latest checkpoint file.")
        # checkpoint_path = find_checkpoint(log_dir.parent, checkpoint_file)
        # Check if we found a checkpoint file
        logging.info(f"Loading checkpoint at {checkpoint}")
        study = optuna.load_study(
            study_name="tuning",
            storage="sqlite:///" + str(checkpoint),
            sampler=sampler,
            pruner=pruner,
        )
        n_calls = n_calls - len(study.trials)
    else:
        if checkpoint:
            logging.warning("Checkpoint path given as flag but not found, starting from scratch.")
        study = optuna.create_study(
            sampler=sampler,
            storage="sqlite:///" + str(log_dir / checkpoint_file),
            study_name="tuning",
            pruner=pruner,
            load_if_exists=True,
        )

    callbacks = [tune_step_callback]
    if wandb:
        wandb_kwargs = {
            "config": {"sampler": sampler},
            "allow_val_change": True,
        }
        wandbc = WeightsAndBiasesCallback(metric_name="loss", wandb_kwargs=wandb_kwargs)
        callbacks.append(wandbc)

    logging.info(f"Starting or resuming Optuna study with {n_calls} trails and callbacks: {callbacks}.")
    if n_calls > 0:
        study.optimize(
            lambda trail: objective(trail, hyperparams_bounds, hyperparams_names),
            n_trials=n_calls,
            callbacks=callbacks,
            gc_after_trial=True,
        )
    else:
        logging.info("No more hyperparameter tuning iterations left, skipping tuning.")
        logging.info("Training with these hyperparameters:")
        bind_gin_params(study.best_params)
        return
    logging.disable(level=NOTSET)

    if do_tune:
        log_full_line("FINISHED TUNING", level=TUNE, char="=", num_newlines=4)

    logging.info("Training with these hyperparameters:")
    bind_gin_params(study.best_params)

    # Without tuning, the study holds a single untrained trial: there is nothing to plot
    if plot and do_tune:
        logging.info("Plotting hyperparameter importances.")
        # The optuna.visualization.matplotlib plots: the default optuna.visualization ones are plotly figures.
        # They set the ggplot style globally, which rc_context restores afterwards
        plots = {"param_importances": plot_param_importances, "optimization_history": plot_optimization_history}
        with plt.rc_context():
            for name, plot_study in plots.items():
                # Each plot on its own: the importances need two completed trials, the history only one
                try:
                    figure = plot_study(study).figure
                    figure.savefig(log_dir / f"{name}.png", bbox_inches="tight")
                except Exception as e:
                    logging.error(f"Failed to plot {name}: {e}")
                finally:
                    plt.close("all")

make_sampler

make_sampler(sampler: type, seed: int, n_initial_points: int) -> BaseSampler

Instantiates the sampler class; the GP sampler explores n_initial_points random trials first.

Source code in icu_benchmarks/tuning/hyperparameters.py
def make_sampler(sampler: type, seed: int, n_initial_points: int) -> optuna.samplers.BaseSampler:
    """Instantiates the sampler class; the GP sampler explores n_initial_points random trials first."""
    if isinstance(sampler, type) and issubclass(sampler, optuna.samplers.GPSampler):
        return sampler(seed=seed, n_startup_trials=n_initial_points, deterministic_objective=True)
    return sampler(seed=seed)

suggest_hyperparameter

suggest_hyperparameter(trial: Trial, name: str, value)

Samples one hyperparameter from its gin bound.

Tuples of numbers are ranges: (low, high), optionally followed by a prior ("log" or "log-uniform") and a log base, which is ignored because the base does not change a log-uniform distribution. Anything else is a list of choices.

Source code in icu_benchmarks/tuning/hyperparameters.py
def suggest_hyperparameter(trial: optuna.trial.Trial, name: str, value):
    """Samples one hyperparameter from its gin bound.

    Tuples of numbers are ranges: (low, high), optionally followed by a prior ("log" or "log-uniform") and a log base,
    which is ignored because the base does not change a log-uniform distribution. Anything else is a list of choices.
    """
    numeric = isinstance(value, tuple) and all(isinstance(v, (int, float)) and not isinstance(v, bool) for v in value[:2])
    if not numeric:
        return trial.suggest_categorical(name, value)
    if len(value) > 2 and not isinstance(value[2], str):
        raise ValueError(f"Hyperparameter {name} = {value} is ambiguous; use a list for choices or (low, high, prior).")
    log = len(value) > 2 and value[2].lower() in LOG_PRIORS
    if isinstance(value[0], int) and isinstance(value[1], int):
        return trial.suggest_int(name, value[0], value[1], log=log)
    return trial.suggest_float(name, value[0], value[1], log=log)

find_checkpoint

find_checkpoint(log_dir, checkpoint_file)

Find the latest checkpoint in the log directory.

Source code in icu_benchmarks/tuning/hyperparameters.py
def find_checkpoint(log_dir, checkpoint_file):
    """Find the latest checkpoint in the log directory."""
    hyperparameters = sorted(log_dir.glob(f"*/{checkpoint_file}"), reverse=True)
    if not hyperparameters:
        return None
    return hyperparameters[0]

icu_benchmarks.tuning.gin_utils

Functions:

Name Description
get_gin_hyperparameters

Get hyperparameters to tune from gin config.

bind_gin_params

Binds hyperparameter dict to gin config and logs them.

get_gin_hyperparameters

get_gin_hyperparameters(class_to_tune: str = REQUIRED, **hyperparams: dict) -> dict

Get hyperparameters to tune from gin config.

Hyperparameters that are already present in the gin config are ignored. Hyperparameters that are not a list or tuple are bound directly to the class. Hyperparameters that are a list or tuple are returned to be tuned.

Parameters:

Name Type Description Default
class_to_tune str

Name of the class to tune hyperparameters for.

REQUIRED
**hyperparams dict

Dictionary of hyperparameters to potentially tune.

{}

Returns:

Type Description
dict

Dictionary of hyperparameters to tune.

Source code in icu_benchmarks/tuning/gin_utils.py
@gin.configurable("hyperparameter")
def get_gin_hyperparameters(class_to_tune: str = gin.REQUIRED, **hyperparams: dict) -> dict:
    """Get hyperparameters to tune from gin config.

    Hyperparameters that are already present in the gin config are ignored.
    Hyperparameters that are not a list or tuple are bound directly to the class.
    Hyperparameters that are a list or tuple are returned to be tuned.

    Args:
        class_to_tune: Name of the class to tune hyperparameters for.
        **hyperparams: Dictionary of hyperparameters to potentially tune.

    Returns:
        Dictionary of hyperparameters to tune.
    """
    hyperparams_to_tune = {}
    for param, values in hyperparams.items():
        name = f"{class_to_tune.__name__}.{param}"
        if f"{name}=" in gin.config_str().replace(" ", ""):
            # check if parameter is already bound, directly binding to class always takes precedence
            continue
        if not isinstance(values, (list, tuple)):
            # if parameter is not a tuple, bind it directly
            gin.bind_parameter(name, values)
            continue
        hyperparams_to_tune[name] = values
    return hyperparams_to_tune

bind_gin_params

bind_gin_params(hyperparams: dict[str, any])

Binds hyperparameter dict to gin config and logs them.

Parameters:

Name Type Description Default
hyperparams dict[str, any]

Dictionary of hyperparameters.

required
Source code in icu_benchmarks/tuning/gin_utils.py
def bind_gin_params(hyperparams: dict[str, any]):
    """Binds hyperparameter dict to gin config and logs them.

    Args:
        hyperparams: Dictionary of hyperparameters.
    """
    logging.info("Binding Hyperparameters:")
    for param, value in hyperparams.items():
        gin.bind_parameter(param, value)
        logging.info(f"{param} = {value}")
        wandb_log({param: value})