Skip to content

Yet Another ICU Benchmark

YAIB logo

Yet Another ICU Benchmark (YAIB) is a framework for clinical machine learning experiments on intensive care unit (ICU) and other electronic health record (EHR) data. It takes a cohort of patient stays, preprocesses it, trains and tunes a model with repeated cross-validation, and reports the results. Each step can be configured or replaced.

An experiment combines three parts, each chosen on the command line:

  • A dataset: a cohort in YAIB's parquet format, for example generated from MIMIC, eICU, HiRID or AUMCdb with YAIB-cohorts.
  • A task: binary classification, regression or imputation, set by a task config.
  • A model: one of the built-in models or your own, with its hyperparameters and search space in a model config.

YAIB overview

Get started

Datasets

YAIB supports these ICU datasets out of the box:

Dataset MIMIC-III / IV eICU-CRD HiRID AUMCdb
Admissions 40k / 73k 200k 33k 23k
Version v1.4 / v2.2 v2.0 v1.1.1 v1.0.2
Frequency (time series) 1 hour 5 minutes 2 / 5 minutes up to 1 minute
Originally published 2015 / 2020 2017 2020 2019
Origin USA USA Switzerland Netherlands

Demo cohorts from the openly available MIMIC-III and eICU demo datasets are included in the repository, so you can run YAIB without credentialed access. See Datasets and access.

Tasks

No Task Frequency Type
1 ICU mortality Once per stay (after 24 h) Binary classification
2 Acute kidney injury (AKI) Hourly (within 6 h) Binary classification
3 Sepsis Hourly (within 6 h) Binary classification
4 Kidney function (KF) Once per stay Regression
5 Length of stay (LoS) Hourly (within 7 days) Regression

New tasks can be added; see Adding tasks.

Models

Type Models (-m)
Classical ML LogisticRegression, ElasticNet, RFClassifier, SVMClassifier, SVMRegressor, LGBMClassifier, LGBMRegressor, XGBClassifier, CBClassifier (CatBoost), BRFClassifier and RUSBClassifier (imbalanced-learn)
Deep learning RNN, LSTM, GRU, GRUD (GRU-D), TCN, Transformer, LocalTransformer, BAT (Bi-Axial Transformer), Mamba
Imputation Mean, Median, Zero, MostFrequent, KNN, MICE, ICE, MissForest, GAIN, Miwae, Miracle, Sinkhorn, SoftImpute, MLP, RNN, BRNN, NP, Attention, BRITS, SAITS, CSDI, Diffusion, Simple_Diffusion, DiffWave, SSSDS4, SSSDSA

The model name is the name of its config file in configs/prediction_models/ or configs/imputation_models/.

Citation

If you use YAIB in your research, please cite the ICLR 2024 paper (arXiv:2306.05109):

@inproceedings{vandewaterYetAnotherICUBenchmark2024,
  title = {Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML},
  shorttitle = {Yet Another ICU Benchmark},
  booktitle = {The Twelfth International Conference on Learning Representations},
  author = {van de Water, Robin and Schmidt, Hendrik Nils Aurel and Elbers, Paul and Thoral, Patrick and Arnrich, Bert and Rockenschaub, Patrick},
  year = {2024},
  month = oct,
  urldate = {2024-02-19},
  langid = {english},
}

Acknowledgements

This project has been developed partially under the funding of "Gemeinsamer Bundesausschuss (G-BA) Innovationsausschuss" in the framework of "CASSANDRA - Clinical ASSist AND aleRt Algorithms" (project number 01VSF20015). We would like to acknowledge the work of Alisher Turubayev, Anna Shopova, Fabian Lange, Mahmut Kamalak, Paul Mattes, and Victoria Ayvasky for adding PyTorch Lightning, Weights and Biases compatibility, and several optional imputation methods to a later version of the benchmark repository.

We do not own any of the datasets used in this benchmark. This project uses heavily adapted components of the HiRID benchmark. The demo datasets have been released under an Open Data Commons Open Database License (ODbL).