Weights & Biases¶
YAIB integrates with Weights & Biases (W&B) to track runs and to run grids of experiments as
sweeps. You need a W&B account; log in once with uv run wandb login.
Sweeps¶
A sweep runs icu-benchmarks once for every combination of parameters. The sweep files in experiments/ reproduce
the paper's benchmarks; demo_benchmark_classification.yml runs the demo cohorts:
command:
- ${env}
- ${program}
- train
- -d
- ../data/
- -t
- BinaryClassification
- --log-dir
- ../yaib_logs
- --tune
- --wandb-sweep
- -gc
- -lc
method: grid
name: yaib_demo_classification_benchmark
parameters:
data_dir:
values:
- demo_data/mortality24/eicu_demo
- demo_data/mortality24/mimic_demo
# ...
model:
values:
- LogisticRegression
- LGBMClassifier
- GRU
# ...
Create the sweep and start an agent from the repository root:
uv run wandb sweep --verbose experiments/demo_benchmark_classification.yml
uv run wandb agent <sweep_id>
SLURM
To run each configuration as its own job on a cluster, start agents with uv run wandb agent --count 1 <sweep_id>.
experiments/slurm_base_template.sh is a template for the job script.
How sweep parameters are applied¶
--wandb-sweep (-wd) makes YAIB read the parameters of the current sweep run. Each parameter is applied in two ways:
- If its name is a command-line option (
data_dir,model,seed,task_name, ...), it sets that option. This is how the sweep above varies the dataset and the model. - It is also passed as a gin binding, as with
-hp. So any gin binding can be swept:
parameters:
ImputationPandasDataset.mask_method:
values: [MCAR, MAR, MNAR]
train_common.batch_size:
values: [32, 64]
Runs are named <mode>_<model>_<dataset>, with the source dataset and fine-tuning size added for transfer runs.
With --tune, the tuning trials are logged to W&B too.
More examples are in scripts/sweep_configs/.