Running experiments¶
An experiment in YAIB has three parts, each chosen on the command line:
| Part | Flag | Where it is defined |
|---|---|---|
| The dataset | -d (folder), -n (name) |
A folder with dyn.parquet, sta.parquet and outc.parquet |
| The task | -t (config), -tn (name) |
configs/tasks/<task>.gin: run mode, variables, preprocessing, cross-validation |
| The model | -m |
configs/prediction_models/<model>.gin: the model, its training settings and hyperparameter search space |
The configs are gin files. They bind values to the arguments of YAIB's classes and functions; see Configuration.
Example: an LSTM for mortality prediction¶
uv run icu-benchmarks train \
-d demo_data/mortality24/mimic_demo \
-n mimic_demo \
-t BinaryClassification \
-tn Mortality24 \
-m LSTM \
--seed 2222 \
-l ../yaib_logs/ \
--tune
The task config¶
-t BinaryClassification loads configs/tasks/BinaryClassification.gin:
# COMMON IMPORTS
include "configs/tasks/common/Imports.gin"
# DATASET CONFIGURATION
include "configs/tasks/common/PredictionTaskVariables.gin"
# CROSS-VALIDATION
include "configs/tasks/common/CrossValidation.gin"
# MODE SETTINGS
Run.mode = "Classification"
NUM_CLASSES = 2 # Binary classification
HORIZON = 24
train_common.weight = "balanced"
train_common.ram_cache = True
# DEEP LEARNING
DLPredictionWrapper.loss = @cross_entropy
# SELECTING PREPROCESSOR
preprocess.preprocessor = @base_classification_preprocessor
preprocess.modality_mapping = %modality_mapping
preprocess.vars = %vars
preprocess.use_static = True
# SELECTING DATASET
include "configs/tasks/common/Dataloader.gin"
It sets the run mode, weights the classes for the imbalanced labels, selects the loss for deep learning models and the preprocessor, and includes three shared files:
-
PredictionTaskVariables.ginnames the three data files and the variables and their roles:configs/tasks/common/PredictionTaskVariables.gin (shortened)preprocess.file_names = { "DYNAMIC": "dyn.parquet", "OUTCOME": "outc.parquet", "STATIC": "sta.parquet", } vars = { "GROUP": "stay_id", # one stay per group "SEQUENCE": "time", # the time axis of the dynamic data "LABEL": "label", # the outcome to predict "DYNAMIC": ["alb", "alp", "alt", ..., "urine", "wbc"], "STATIC": ["age", "sex", "height", "weight"], } -
CrossValidation.ginsets 5 repetitions × 5 folds (execute_repeated_cv.cv_repetitionsandcv_folds). -
Dataloader.ginpasses the variables to the datasets.
configs/tasks/Regression.gin is the same for regression; DatasetImputation.gin trains
imputation models.
The model config¶
-m LSTM loads configs/prediction_models/LSTM.gin:
# Settings for Long Short-Term Memory (LSTM) model.
# Common settings for DL models
include "configs/prediction_models/common/DLCommon.gin"
# Train params
train_common.model = @LSTMNet
# Optimizer params
optimizer/hyperparameter.class_to_tune = @Adam
optimizer/hyperparameter.weight_decay = 1e-6
optimizer/hyperparameter.lr = (1e-5, 3e-4)
# Encoder params
model/hyperparameter.class_to_tune = @LSTMNet
model/hyperparameter.num_classes = %NUM_CLASSES
model/hyperparameter.hidden_dim = (32, 256, "log-uniform", 2)
model/hyperparameter.layer_dim = (1, 3)
DLCommon.gin holds the settings shared by all deep learning models: no feature generation in preprocessing, the Adam
optimizer, up to 50 epochs with early stopping after 10 epochs without improvement, batch size 64, and the tuning
budget (DLTuning.gin). Classical models include MLCommon.gin instead, which turns on feature generation (summary
statistics over time) for the dynamic variables.
The hyperparameter bindings define the search space: a tuple (low, high) is a range, a list is a set of choices, and
a single value is fixed. See Hyperparameter tuning.
What a run does¶
- Configure: parse the task and model configs, then apply
-hpoverrides. - Tune (with
--tune): search the hyperparameters with Optuna on the first folds, then bind the best ones. - Cross-validate: for each repetition and fold, split the stays into train, validation and test sets, preprocess them (fitting the preprocessing on the training split only), train the model, and evaluate it on the test split.
- Aggregate: collect the metrics of all folds, with their mean and confidence interval.
Results go to <log-dir>/<name>/<task-name>/<model>/<timestamp>/; the Quickstart
shows the files.
Caching the preprocessed data¶
Preprocessing the full datasets takes time. With -gc (--generate_cache) YAIB saves the preprocessed splits next to
the data, in <data-dir>/cache/, and with -lc (--load_cache) it reuses them. The cache key includes the seed,
fold, task settings and preprocessing configuration, so a changed configuration does not load a stale cache. Use both
flags together to create the cache on the first run and reuse it afterwards.
Useful options¶
| Option | Effect |
|---|---|
-hp <binding> ... |
Override any gin binding, e.g. -hp train_common.epochs=10. |
--cpu |
Train on the CPU even if a GPU is available. |
-db / --debug |
Load only a small part of the data, for quick tests. |
--reproducible |
Make PyTorch deterministic. |
--complete-train |
Train on all the data, without a test split. |
-mo DYNAMIC / --modalities |
Train on a subset of the modalities (keys of modality_mapping). |
--label <name> |
Choose one label when the task config lists several. |
-pl / --plot |
Save common plots of the results. |
--explain-features |
Save feature attributions: SHAP for tree-based models, Integrated Gradients and more for deep learning models; see Feature explanations. |
The CLI reference lists every option.