Skip to content

Adding datasets

YAIB reads any cohort in its data format, so a new dataset needs no code changes once it is in that format. The YAIB-cohorts code produces exactly this format for the supported datasets.

The data format

A cohort is a folder with three Apache Parquet files:

File Contents Rows
dyn.parquet Dynamic variables that change during the stay (vital signs, lab values, ...) One per stay and time step
sta.parquet Static variables (age, sex, height, weight, ...) One per stay
outc.parquet The label to predict One per stay, or one per stay and time step for hourly tasks

The task config tells YAIB which file is which and what role each column has:

configs/tasks/common/PredictionTaskVariables.gin (shortened)
preprocess.file_names = {
    "DYNAMIC": "dyn.parquet",
    "OUTCOME": "outc.parquet",
    "STATIC": "sta.parquet",
}

vars = {
    "GROUP": "stay_id",
    "SEQUENCE": "time",
    "LABEL": "label",
    "DYNAMIC": ["alb", "alp", "alt", ..., "wbc"],
    "STATIC": ["age", "sex", "height", "weight"],
}
Role Meaning
GROUP The ID column that groups rows into stays; it is in all three files.
SEQUENCE The time column of the dynamic data (and of hourly labels).
LABEL The label column in outc.parquet. It can be a list of labels; choose one with --label.
DYNAMIC The dynamic feature columns.
STATIC The static feature columns.

The roles come from ReciPys, the preprocessing package developed alongside YAIB.

If your files or columns are named differently, copy PredictionTaskVariables.gin and the task config, change the names, and select your task config with -t; see Adding tasks.

Harmonizing a new data source

To use the existing clinical concepts and task definitions with a new database, make it available to ricu, which YAIB-cohorts uses to harmonize the datasets. The example below follows the integration of SICdb.

Define ID types and table structure

ricu needs to know which tables and columns exist in the data source. This is specified in a JSON configuration file, partially shown below:

{
  "name": "sic",
  "id_cfg": {
    "patient": {
      "id": "patientid",
      "position": 1,
      "start": "firstadmission",
      "end": "offsetofdeath",
      "table": "cases"
    },
    "icustay": {
      "id": "caseid",
      "position": 2,
      "start": "offsetafterfirstadmission",
      "end": "timeofstay",
      "table": "cases"
    }
  },
  "tables": {
    "cases": {
      "files": "cases.csv.gz",
      "defaults": {
        "index_var": "offsetafterfirstadmission",
        "time_vars": [
          "offsetafterfirstadmission",
          "offsetofdeath"
        ]
      },
      "cols": {
        "caseid": {
          "name": "CaseID",
          "spec": "col_integer"
        },
        "patientid": {
          "name": "PatientID",
          "spec": "col_integer"
        },
        "admissionyear": {
          "name": "AdmissionYear",
          "spec": "col_integer"
        }
      }
    }
  }
}

Tables are defined under tables. Each table lists its source file (usually .csv or .csv.gz) and its columns with their data types. Default roles can be set for some columns, usually the time index, the other time columns, and the column with the value of interest.

ricu also needs the main ID types of the dataset. Information in ICU datasets is linked to a unit of observation: the patient, the hospital admission (hospadm) or the ICU stay (icustay). Knowing which ID each piece of information belongs to lets ricu relate all information in time. For example, MIMIC-IV records labevents per hospital admission and chartevents per ICU stay; defining how the two ID systems relate maps both to a common time scale, such as the time since ICU admission. Knowing that an ICU stay is described in icustays in MIMIC and in cases in SICdb defines the same reference point in both databases.

SICdb has two ID types, the patient and the ICU stay, and no separate hospital admission. A patient is observed from their first admission until their death (if it occurred); an ICU stay lasts from admission to the ICU until discharge.

Calculate origin times for each ID type

ricu also needs to know when each ID starts and ends. Much of this is automated, but small adjustments are often needed. SICdb, for example, gives discharge times in seconds since admission rather than as absolute times. Such adjustments are made by subtyping the respective function (here id_win_helper) and overriding its default behavior. ricu's helper functions also set up relative times in seconds.

With these steps, SICdb is available in ricu. Further helpers can add optional functionality, such as finding the measurement units, which SICdb stores in a separate reference table. This only interfaces the database: each clinical concept still has to be mapped to the database's items, for example through their measurement IDs; see Clinical concepts. ricu's helpers make this mapping much simpler. More on ricu and its design is in Bennett et al. (2023).