Skip to content

Running experiments

An experiment in YAIB has three parts, each chosen on the command line:

Part Flag Where it is defined
The dataset -d (folder), -n (name) A folder with dyn.parquet, sta.parquet and outc.parquet
The task -t (config), -tn (name) configs/tasks/<task>.gin: run mode, variables, preprocessing, cross-validation
The model -m configs/prediction_models/<model>.gin: the model, its training settings and hyperparameter search space

The configs are gin files. They bind values to the arguments of YAIB's classes and functions; see Configuration.

Example: an LSTM for mortality prediction

uv run icu-benchmarks train \
    -d demo_data/mortality24/mimic_demo \
    -n mimic_demo \
    -t BinaryClassification \
    -tn Mortality24 \
    -m LSTM \
    --seed 2222 \
    -l ../yaib_logs/ \
    --tune

The task config

-t BinaryClassification loads configs/tasks/BinaryClassification.gin:

configs/tasks/BinaryClassification.gin
# COMMON IMPORTS
include "configs/tasks/common/Imports.gin"

# DATASET CONFIGURATION
include "configs/tasks/common/PredictionTaskVariables.gin"

# CROSS-VALIDATION
include "configs/tasks/common/CrossValidation.gin"

# MODE SETTINGS
Run.mode = "Classification"
NUM_CLASSES = 2 # Binary classification
HORIZON = 24
train_common.weight = "balanced"
train_common.ram_cache = True

# DEEP LEARNING
DLPredictionWrapper.loss = @cross_entropy

# SELECTING PREPROCESSOR
preprocess.preprocessor = @base_classification_preprocessor
preprocess.modality_mapping = %modality_mapping
preprocess.vars = %vars
preprocess.use_static = True

# SELECTING DATASET
include "configs/tasks/common/Dataloader.gin"

It sets the run mode, weights the classes for the imbalanced labels, selects the loss for deep learning models and the preprocessor, and includes three shared files:

  • PredictionTaskVariables.gin names the three data files and the variables and their roles:

    configs/tasks/common/PredictionTaskVariables.gin (shortened)
    preprocess.file_names = {
        "DYNAMIC": "dyn.parquet",
        "OUTCOME": "outc.parquet",
        "STATIC": "sta.parquet",
    }
    
    vars = {
        "GROUP": "stay_id",  # one stay per group
        "SEQUENCE": "time",  # the time axis of the dynamic data
        "LABEL": "label",  # the outcome to predict
        "DYNAMIC": ["alb", "alp", "alt", ..., "urine", "wbc"],
        "STATIC": ["age", "sex", "height", "weight"],
    }
    
  • CrossValidation.gin sets 5 repetitions × 5 folds (execute_repeated_cv.cv_repetitions and cv_folds).

  • Dataloader.gin passes the variables to the datasets.

configs/tasks/Regression.gin is the same for regression; DatasetImputation.gin trains imputation models.

The model config

-m LSTM loads configs/prediction_models/LSTM.gin:

configs/prediction_models/LSTM.gin
# Settings for Long Short-Term Memory (LSTM) model.

# Common settings for DL models
include "configs/prediction_models/common/DLCommon.gin"

# Train params
train_common.model = @LSTMNet

# Optimizer params
optimizer/hyperparameter.class_to_tune = @Adam
optimizer/hyperparameter.weight_decay = 1e-6
optimizer/hyperparameter.lr = (1e-5, 3e-4)

# Encoder params
model/hyperparameter.class_to_tune = @LSTMNet
model/hyperparameter.num_classes = %NUM_CLASSES
model/hyperparameter.hidden_dim = (32, 256, "log-uniform", 2)
model/hyperparameter.layer_dim = (1, 3)

DLCommon.gin holds the settings shared by all deep learning models: no feature generation in preprocessing, the Adam optimizer, up to 50 epochs with early stopping after 10 epochs without improvement, batch size 64, and the tuning budget (DLTuning.gin). Classical models include MLCommon.gin instead, which turns on feature generation (summary statistics over time) for the dynamic variables.

The hyperparameter bindings define the search space: a tuple (low, high) is a range, a list is a set of choices, and a single value is fixed. See Hyperparameter tuning.

What a run does

  1. Configure: parse the task and model configs, then apply -hp overrides.
  2. Tune (with --tune): search the hyperparameters with Optuna on the first folds, then bind the best ones.
  3. Cross-validate: for each repetition and fold, split the stays into train, validation and test sets, preprocess them (fitting the preprocessing on the training split only), train the model, and evaluate it on the test split.
  4. Aggregate: collect the metrics of all folds, with their mean and confidence interval.

Results go to <log-dir>/<name>/<task-name>/<model>/<timestamp>/; the Quickstart shows the files.

Caching the preprocessed data

Preprocessing the full datasets takes time. With -gc (--generate_cache) YAIB saves the preprocessed splits next to the data, in <data-dir>/cache/, and with -lc (--load_cache) it reuses them. The cache key includes the seed, fold, task settings and preprocessing configuration, so a changed configuration does not load a stale cache. Use both flags together to create the cache on the first run and reuse it afterwards.

Useful options

Option Effect
-hp <binding> ... Override any gin binding, e.g. -hp train_common.epochs=10.
--cpu Train on the CPU even if a GPU is available.
-db / --debug Load only a small part of the data, for quick tests.
--reproducible Make PyTorch deterministic.
--complete-train Train on all the data, without a test split.
-mo DYNAMIC / --modalities Train on a subset of the modalities (keys of modality_mapping).
--label <name> Choose one label when the task config lists several.
-pl / --plot Save common plots of the results.
--explain-features Save feature attributions: SHAP for tree-based models, Integrated Gradients and more for deep learning models; see Feature explanations.

The CLI reference lists every option.