Data¶
Loading, splitting and preprocessing the cohorts, and the datasets the models train on.
icu_benchmarks.data.split_process_data
¶
Functions:
| Name | Description |
|---|---|
select_label |
The label to train on: the requested one, which must be configured, else the first of several configured labels. |
preprocess_data |
Perform loading, splitting, imputing and normalising of task data. |
check_sanitize_data |
Check for duplicates in the loaded data and remove them. |
make_train_val_pandas |
Randomly splits the data into training and validation sets for fitting a full model, |
join_id_dtype |
The dtype all segments' IDs are cast to for the split joins: integer IDs are widened, others (e.g. strings) kept. |
make_train_val_polars |
Randomly splits the data into training and validation sets for fitting a full model, |
make_train_val |
Randomly splits the data into training and validation sets for fitting a full model. |
make_single_split_pandas |
Randomly splits the data into training, validation, and test sets, |
make_single_split_polars |
Randomly splits the data into training, validation, and test set, |
make_single_split |
Randomly splits the data into training, validation, and test sets. |
check_required_keys |
Checks if all required keys are present in the vars dictionary. |
select_label
¶
The label to train on: the requested one, which must be configured, else the first of several configured labels.
Source code in icu_benchmarks/data/split_process_data.py
preprocess_data
¶
preprocess_data(data_dir: Path, file_names: dict[str, str] | Any = REQUIRED, preprocessor: type[PolarsClassificationPreprocessor | PolarsRegressionPreprocessor] = PolarsClassificationPreprocessor, use_static: bool = True, vars: dict[str, str | list[str]] | Any = REQUIRED, modality_mapping: Optional[dict[str, list[str]]] = None, selected_modalities: Optional[list[str]] = None, seed: int = 42, debug: bool = False, cv_repetitions: int = 5, repetition_index: int = 0, cv_folds: int = 5, train_size: Optional[float] = None, load_cache: bool = False, generate_cache: bool = False, fold_index: int = 0, pretrained_imputation_model: Optional[str] = None, complete_train: bool = False, runmode: RunMode = classification, label: Optional[str] = None, required_var_types: Optional[list[str]] = None, required_segments: Optional[list[str]] = None, target_transform: Optional[TargetTransform] = None) -> tuple[dict[str, dict[str, DataFrame]], Optional[TargetTransform]]
Perform loading, splitting, imputing and normalising of task data.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
use_static
|
bool
|
Whether to use static features (for DL models). |
True
|
complete_train
|
bool
|
Whether to use all data for training/validation. |
False
|
runmode
|
RunMode
|
Run mode. Can be one of the values of RunMode |
classification
|
preprocessor
|
type[PolarsClassificationPreprocessor | PolarsRegressionPreprocessor]
|
Define the preprocessor. |
PolarsClassificationPreprocessor
|
data_dir
|
Path
|
Path to the directory holding the data. |
required |
file_names
|
dict[str, str] | Any
|
Contains the parquet file names in data_dir. |
REQUIRED
|
vars
|
dict[str, str | list[str]] | Any
|
Contains the names of columns in the data. |
REQUIRED
|
seed
|
int
|
Random seed. |
42
|
debug
|
bool
|
Load less data if true. |
False
|
cv_repetitions
|
int
|
Number of times to repeat cross validation. |
5
|
repetition_index
|
int
|
Index of the repetition to return. |
0
|
cv_folds
|
int
|
Number of folds to use for cross validation. |
5
|
train_size
|
Optional[float]
|
Fixed size of train split (including validation data). |
None
|
load_cache
|
bool
|
Use cached preprocessed data if true. |
False
|
generate_cache
|
bool
|
Generate cached preprocessed data if true. |
False
|
fold_index
|
int
|
Index of the fold to return. |
0
|
pretrained_imputation_model
|
Optional[str]
|
pretrained imputation model to use. if None, standard imputation is used. |
None
|
Returns:
| Type | Description |
|---|---|
dict[str, dict[str, DataFrame]]
|
(Preprocessed split data, fitted regression target transform or None). |
Optional[TargetTransform]
|
A supplied target_transform is reused without fitting (evaluation/fine-tuning). |
Source code in icu_benchmarks/data/split_process_data.py
46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 | |
check_sanitize_data
¶
check_sanitize_data(data: dict[str, DataFrame], vars: dict[str, str | list[str]]) -> dict[str, DataFrame]
Check for duplicates in the loaded data and remove them.
Source code in icu_benchmarks/data/split_process_data.py
make_train_val_pandas
¶
make_train_val_pandas(data: dict[str, DataFrame], vars: dict[str, Union[str, list[str]]], train_size: Optional[float] = 0.8, seed: int = 42, debug: bool = False, runmode: RunMode = classification) -> dict[str, dict[str, DataFrame]]
Randomly splits the data into training and validation sets for fitting a full model, specifically designed for Pandas DataFrames.
For a more detailed documentation refer to make_train_val(...)
Source code in icu_benchmarks/data/split_process_data.py
join_id_dtype
¶
The dtype all segments' IDs are cast to for the split joins: integer IDs are widened, others (e.g. strings) kept.
Integer IDs in one file and string IDs in another would not match after the cast ("001" is not "1"), so the join would silently drop those stays' features; float and string IDs can also collapse ("1" and "01" are both 1.0). Such mixes are rejected instead.
Source code in icu_benchmarks/data/split_process_data.py
make_train_val_polars
¶
make_train_val_polars(data: dict[str, DataFrame], vars: dict[str, Union[str, list[str]]], train_size: Optional[float] = 0.8, seed: int = 42, debug: bool = False, runmode: RunMode = classification) -> dict[str, dict[str, DataFrame]]
Randomly splits the data into training and validation sets for fitting a full model, specifically designed for Polars DataFrames.
For a more detailed documentation refer to make_train_val(...)
Source code in icu_benchmarks/data/split_process_data.py
make_train_val
¶
make_train_val(data: dict[str, Union[DataFrame, DataFrame]], vars: dict[str, Union[str, list[str]]], train_size: Optional[float] = 0.8, seed: int = 42, debug: bool = False, runmode: RunMode = classification, polars: bool = True) -> dict[str, dict[str, DataFrame]] | dict[str, dict[str, DataFrame]]
Randomly splits the data into training and validation sets for fitting a full model. Dispatches to either a Polars or Pandas backend based on the 'polars' flag.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, Union[DataFrame, DataFrame]]
|
A dictionary containing DataFrames (either Pandas or Polars), divided into segments like OUTCOME, STATIC, and DYNAMIC. |
required |
vars
|
dict[str, Union[str, list[str]]]
|
A dictionary containing the names of columns (variables) in the data. |
required |
train_size
|
Optional[float]
|
The proportion of the dataset to include in the train split. |
0.8
|
seed
|
int
|
Random seed for reproducibility. |
42
|
debug
|
bool
|
If True, uses only a small fraction (1%) of the data for debugging. |
False
|
runmode
|
RunMode
|
The type of machine learning task (e.g., classification, regression). |
classification
|
polars
|
bool
|
If True, uses the Polars backend; otherwise, uses the Pandas backend. |
True
|
Returns:
| Type | Description |
|---|---|
dict[str, dict[str, DataFrame]] | dict[str, dict[str, DataFrame]]
|
A dictionary containing the input data divided into 'train', 'val', and 'test' |
dict[str, dict[str, DataFrame]] | dict[str, dict[str, DataFrame]]
|
splits. Each split is itself a dictionary of DataFrames (Pandas or Polars) |
dict[str, dict[str, DataFrame]] | dict[str, dict[str, DataFrame]]
|
corresponding to the original data segments. |
Source code in icu_benchmarks/data/split_process_data.py
make_single_split_pandas
¶
make_single_split_pandas(data: dict[str, DataFrame], vars: dict[str, Union[str, list[str]]], cv_repetitions: int, repetition_index: int, cv_folds: int, fold_index: int, train_size: Optional[float] = None, seed: int = 42, debug: bool = False, runmode: RunMode = classification) -> dict[str, dict[str, DataFrame]]
Randomly splits the data into training, validation, and test sets, specifically designed for Pandas DataFrames.
For a more detailed documentation refer to make_single_splits(...)
Source code in icu_benchmarks/data/split_process_data.py
521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 | |
make_single_split_polars
¶
make_single_split_polars(data: dict[str, DataFrame], vars: dict[str, Union[str, list[str]]], cv_repetitions: int, repetition_index: int, cv_folds: int, fold_index: int, train_size: Optional[float] = None, seed: int = 42, debug: bool = False, runmode: RunMode = classification) -> dict[str, dict[str, DataFrame]]
Randomly splits the data into training, validation, and test set, specifically designed for Polars DataFrames.
For a more detailed documentation refer to make_single_splits(...)
Source code in icu_benchmarks/data/split_process_data.py
600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 | |
make_single_split
¶
make_single_split(data: dict[str, Union[DataFrame, DataFrame]], vars: dict[str, Union[str, list[str]]], cv_repetitions: int, repetition_index: int, cv_folds: int, fold_index: int, train_size: Optional[float] = None, seed: int = 42, debug: bool = False, runmode: RunMode = classification, polars: bool = True) -> dict[str, dict[str, DataFrame]] | dict[str, dict[str, DataFrame]]
Randomly splits the data into training, validation, and test sets. Dispatches to either a Polars or Pandas backend based on the 'polars' flag.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, Union[DataFrame, DataFrame]]
|
A dictionary containing DataFrames (either Pandas or Polars), divided into segments like OUTCOME, STATIC, and DYNAMIC. |
required |
vars
|
dict[str, Union[str, list[str]]]
|
Contains the names of columns in the data. |
required |
cv_repetitions
|
int
|
Number of times to repeat cross validation. |
required |
repetition_index
|
int
|
Index of the repetition to return. |
required |
cv_folds
|
int
|
Number of folds for cross validation. |
required |
fold_index
|
int
|
Index of the fold to return. |
required |
train_size
|
Optional[float]
|
Fixed size of train split (including validation data). |
None
|
seed
|
int
|
Random seed. |
42
|
debug
|
bool
|
Load less data if true. |
False
|
runmode
|
RunMode
|
Run mode. Can be one of the values of RunMode |
classification
|
polars
|
bool
|
If True, uses the Polars backend; otherwise, uses the Pandas backend. |
True
|
Returns:
| Type | Description |
|---|---|
dict[str, dict[str, DataFrame]] | dict[str, dict[str, DataFrame]]
|
Input data divided into 'train', 'val', and 'test'. |
Source code in icu_benchmarks/data/split_process_data.py
check_required_keys
¶
Checks if all required keys are present in the vars dictionary.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
vars
|
dict
|
The dictionary to check. |
required |
required_keys
|
list
|
The list of required keys. |
required |
Raises:
| Type | Description |
|---|---|
KeyError
|
If any required key is missing. |
Source code in icu_benchmarks/data/split_process_data.py
icu_benchmarks.data.preprocessor
¶
Classes:
| Name | Description |
|---|---|
PolarsClassificationPreprocessor |
|
PolarsRegressionPreprocessor |
|
PandasClassificationPreprocessor |
|
PandasRegressionPreprocessor |
|
PandasImputationPreprocessor |
Preprocesses data for imputation. |
Functions:
| Name | Description |
|---|---|
apply_recipe_to_splits |
Fits and transforms the training features, then transforms the validation and test features with the recipe. |
cache_recipe |
Cache recipe to make it available for e.g. transfer learning. |
restore_recipe |
Restore recipe from cache to use for e.g. transfer learning. |
PolarsClassificationPreprocessor
¶
PolarsClassificationPreprocessor(generate_features: bool = False, scaling: bool = True, use_static_features: bool = True, save_cache: Optional[Union[str, Path]] = None, load_cache: Optional[Union[str, Path]] = None, vars_to_exclude: Optional[list[str]] = None)
Bases: Preprocessor
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
generate_features
|
bool
|
Generate features for dynamic data. |
False
|
scaling
|
bool
|
Scaling of dynamic and static data. |
True
|
use_static_features
|
bool
|
Use static features. |
True
|
save_cache
|
Optional[Union[str, Path]]
|
Save recipe cache from this path. |
None
|
load_cache
|
Optional[Union[str, Path]]
|
Load recipe cache from this path. |
None
|
vars_to_exclude
|
Optional[list[str]]
|
Variables to exclude from missing indicator/ feature generation. |
None
|
Returns: Preprocessed data.
Methods:
| Name | Description |
|---|---|
apply |
Args: |
Source code in icu_benchmarks/data/preprocessor.py
apply
¶
apply(data: dict[str, dict[str, DataFrame]], vars: dict[str, Union[str, list[str]]]) -> dict[str, dict[str, DataFrame]]
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, dict[str, DataFrame]]
|
Train, validation and test data dictionary. Further divided in static, dynamic, and outcome. |
required |
vars
|
dict[str, Union[str, list[str]]]
|
Variables for static, dynamic, outcome. |
required |
Returns: Preprocessed data.
Source code in icu_benchmarks/data/preprocessor.py
PolarsRegressionPreprocessor
¶
PolarsRegressionPreprocessor(generate_features: bool = False, scaling: bool = True, use_static_features: bool = True, outcome_max=None, outcome_min=None, save_cache: Optional[Union[str, Path]] = None, load_cache: Optional[Union[str, Path]] = None, vars_to_exclude: Optional[list[str]] = None, target_scaling=None)
Bases: PolarsClassificationPreprocessor
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
generate_features
|
bool
|
Generate features for dynamic data. |
False
|
scaling
|
bool
|
Scaling of dynamic and static data. |
True
|
use_static_features
|
bool
|
Use static features. |
True
|
max_range
|
Maximum value in outcome. |
required | |
min_range
|
Minimum value in outcome. |
required | |
save_cache
|
Optional[Union[str, Path]]
|
Save recipe cache. |
None
|
load_cache
|
Optional[Union[str, Path]]
|
Load recipe cache. |
None
|
Returns: Preprocessed data.
Methods:
| Name | Description |
|---|---|
apply |
Args: |
Source code in icu_benchmarks/data/preprocessor.py
apply
¶
apply(data: dict[str, dict[str, DataFrame]], vars: dict[str, Union[str, list[str]]]) -> dict[str, dict[str, DataFrame]]
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, dict[str, DataFrame]]
|
Train, validation and test data dictionary. Further divided in static, dynamic, and outcome. |
required |
vars
|
dict[str, Union[str, list[str]]]
|
Variables for static, dynamic, outcome. |
required |
Returns: Preprocessed data.
Source code in icu_benchmarks/data/preprocessor.py
PandasClassificationPreprocessor
¶
PandasClassificationPreprocessor(generate_features: bool = True, scaling: bool = True, use_static_features: bool = True, save_cache: Optional[Union[str, Path]] = None, load_cache: Optional[Union[str, Path]] = None)
Bases: Preprocessor
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
generate_features
|
bool
|
Generate features for dynamic data. |
True
|
scaling
|
bool
|
Scaling of dynamic and static data. |
True
|
use_static_features
|
bool
|
Use static features. |
True
|
save_cache
|
Optional[Union[str, Path]]
|
Save recipe cache from this path. |
None
|
load_cache
|
Optional[Union[str, Path]]
|
Load recipe cache from this path. |
None
|
Returns: Preprocessed data.
Methods:
| Name | Description |
|---|---|
apply |
Args: |
Source code in icu_benchmarks/data/preprocessor.py
apply
¶
apply(data: dict[str, dict[str, DataFrame]], vars: dict[str, Union[str, list[str]]]) -> dict[str, dict[str, DataFrame]]
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, dict[str, DataFrame]]
|
Train, validation and test data dictionary. Further divided in static, dynamic, and outcome. |
required |
vars
|
dict[str, Union[str, list[str]]]
|
Variables for static, dynamic, outcome. |
required |
Returns: Preprocessed data.
Source code in icu_benchmarks/data/preprocessor.py
PandasRegressionPreprocessor
¶
PandasRegressionPreprocessor(generate_features: bool = True, scaling: bool = True, use_static_features: bool = True, outcome_max=None, outcome_min=None, save_cache: Optional[Union[str, Path]] = None, load_cache: Optional[Union[str, Path]] = None, target_scaling=None)
Bases: PandasClassificationPreprocessor
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
generate_features
|
bool
|
Generate features for dynamic data. |
True
|
scaling
|
bool
|
Scaling of dynamic and static data. |
True
|
use_static_features
|
bool
|
Use static features. |
True
|
max_range
|
Maximum value in outcome. |
required | |
min_range
|
Minimum value in outcome. |
required | |
save_cache
|
Optional[Union[str, Path]]
|
Save recipe cache. |
None
|
load_cache
|
Optional[Union[str, Path]]
|
Load recipe cache. |
None
|
Returns: Preprocessed data.
Methods:
| Name | Description |
|---|---|
apply |
Args: |
Source code in icu_benchmarks/data/preprocessor.py
apply
¶
apply(data: dict[str, dict[str, DataFrame]], vars: dict[str, Union[str, list[str]]]) -> dict[str, dict[str, DataFrame]]
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, dict[str, DataFrame]]
|
Train, validation and test data dictionary. Further divided in static, dynamic, and outcome. |
required |
vars
|
dict[str, Union[str, list[str]]]
|
Variables for static, dynamic, outcome. |
required |
Returns: Preprocessed data.
Source code in icu_benchmarks/data/preprocessor.py
PandasImputationPreprocessor
¶
PandasImputationPreprocessor(scaling: bool = True, use_static_features: bool = True, filter_missing_values: bool = True, save_cache: Optional[Union[str, Path]] = None, load_cache: Optional[Union[str, Path]] = None)
Bases: Preprocessor
Preprocesses data for imputation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scaling
|
bool
|
If the values in each column should be normalized. Defaults to True. |
True
|
use_static_features
|
bool
|
If static features should be included in the dataset. Defaults to True. |
True
|
Methods:
| Name | Description |
|---|---|
apply |
Args: |
Source code in icu_benchmarks/data/preprocessor.py
apply
¶
apply(data: dict[str, dict[str, DataFrame]], vars: dict[str, Union[str, list[str]]]) -> dict[str, dict[str, DataFrame]]
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
dict[str, dict[str, DataFrame]]
|
Train, validation and test data dictionary. Further divided in static, dynamic, and outcome. |
required |
vars
|
dict[str, Union[str, list[str]]]
|
Variables for static, dynamic, outcome. |
required |
Returns: Preprocessed data.
Source code in icu_benchmarks/data/preprocessor.py
apply_recipe_to_splits
¶
apply_recipe_to_splits(recipe: Recipe, data: dict[str, dict[str, Union[DataFrame, DataFrame]]], type: str, save_cache: Optional[Union[str, Path]] = None, load_cache: Optional[Union[str, Path]] = None) -> dict[str, dict[str, Union[DataFrame, DataFrame]]]
Fits and transforms the training features, then transforms the validation and test features with the recipe. Works with both Polars and Pandas versions of recipys.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
load_cache
|
Optional[Union[str, Path]]
|
Load recipe from cache, for e.g. transfer learning. |
None
|
save_cache
|
Optional[Union[str, Path]]
|
Save recipe to cache, for e.g. transfer learning. |
None
|
recipe
|
Recipe
|
Object containing info about the features and steps. |
required |
data
|
dict[str, dict[str, Union[DataFrame, DataFrame]]]
|
Dict containing 'train', 'val', and 'test' and types of features per split. |
required |
type
|
str
|
Whether to apply recipe to dynamic features, static features or outcomes. |
required |
Returns:
| Type | Description |
|---|---|
dict[str, dict[str, Union[DataFrame, DataFrame]]]
|
Transformed features divided into 'train', 'val', and 'test'. |
Source code in icu_benchmarks/data/preprocessor.py
cache_recipe
¶
cache_recipe(recipe: Recipe, cache_file: Path) -> None
Cache recipe to make it available for e.g. transfer learning.
Source code in icu_benchmarks/data/preprocessor.py
restore_recipe
¶
restore_recipe(cache_file: Path) -> Recipe
Restore recipe from cache to use for e.g. transfer learning.
Source code in icu_benchmarks/data/preprocessor.py
icu_benchmarks.data.loader
¶
Classes:
| Name | Description |
|---|---|
CommonPolarsDataset |
|
PredictionPolarsDataset |
Subclass of common dataset for prediction tasks. |
CommonPandasDataset |
Common dataset: subclass of Torch Dataset that represents the data to learn on. |
PredictionPandasDataset |
Subclass of common dataset for prediction tasks. |
ImputationPandasDataset |
Subclass of Common Dataset that contains data for imputation models. |
ImputationPredictionDataset |
Subclass of torch dataset that represents data with missingness for imputation. |
Functions:
| Name | Description |
|---|---|
check_class_labels |
Balanced weights are indexed by class, so the labels must be 0..K-1. |
CommonPolarsDataset
¶
CommonPolarsDataset(data: dict, split: str = train, vars: dict[str, Union[str, list[str]]] | object = REQUIRED, grouping_segment: str = outcome, mps: bool = False, name: str = '')
Bases: Dataset
Methods:
| Name | Description |
|---|---|
get_input_names |
The columns of the deep learning input tensors ( |
Source code in icu_benchmarks/data/loader.py
get_input_names
¶
The columns of the deep learning input tensors (self[i][0]), in order.
Unlike get_feature_names in prediction datasets, this keeps every feature column, including counter.
Source code in icu_benchmarks/data/loader.py
PredictionPolarsDataset
¶
PredictionPolarsDataset(*args, ram_cache: bool = True, **kwargs)
Bases: CommonPolarsDataset
Subclass of common dataset for prediction tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ram_cache
|
bool
|
Whether the complete dataset should be stored in ram. Defaults to True. |
True
|
Methods:
| Name | Description |
|---|---|
get_stay_ids |
The stay ID of each item, in item order. |
get_balance |
Return the weight balance for the split of interest. |
get_data_and_labels |
Function to return all the data and labels aligned at once. |
Source code in icu_benchmarks/data/loader.py
get_balance
¶
get_balance() -> list
Return the weight balance for the split of interest.
Returns:
| Type | Description |
|---|---|
list
|
Weights for each label. |
Source code in icu_benchmarks/data/loader.py
get_data_and_labels
¶
get_data_and_labels() -> tuple[ndarray, ndarray, ndarray]
Function to return all the data and labels aligned at once.
We use this function for the ML methods which don't require an iterator.
Returns:
| Type | Description |
|---|---|
tuple[ndarray, ndarray, ndarray]
|
A Tuple containing data points and label for the split. |
Source code in icu_benchmarks/data/loader.py
CommonPandasDataset
¶
CommonPandasDataset(data: dict, split: str = train, vars: dict[str, Union[str, list[str]]] | object = REQUIRED, grouping_segment: str = outcome, mps: bool = False, name: str = '')
Bases: Dataset
Common dataset: subclass of Torch Dataset that represents the data to learn on.
Args: data: Dict of the different splits of the data. split: Either 'train','val' or 'test'. vars: Contains the names of columns in the data. grouping_segment: str, optional: The segment of the data contains the grouping column with only unique values. Defaults to Segment.outcome. Is used to calculate the number of stays in the data.
Source code in icu_benchmarks/data/loader.py
PredictionPandasDataset
¶
PredictionPandasDataset(*args, ram_cache: bool = True, **kwargs)
Bases: CommonPandasDataset
Subclass of common dataset for prediction tasks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ram_cache
|
bool
|
Whether the complete dataset should be stored in ram. Defaults to True. |
True
|
Methods:
| Name | Description |
|---|---|
get_balance |
Return the weight balance for the split of interest. |
get_data_and_labels |
Function to return all the data and labels aligned at once. |
Source code in icu_benchmarks/data/loader.py
get_balance
¶
get_balance() -> list
Return the weight balance for the split of interest.
Returns:
| Type | Description |
|---|---|
list
|
Weights for each label. |
Source code in icu_benchmarks/data/loader.py
get_data_and_labels
¶
get_data_and_labels() -> tuple[ndarray, ndarray, ndarray]
Function to return all the data and labels aligned at once.
We use this function for the ML methods which don't require an iterator.
Returns:
| Type | Description |
|---|---|
tuple[ndarray, ndarray, ndarray]
|
A Tuple containing data points and label for the split. |
Source code in icu_benchmarks/data/loader.py
ImputationPandasDataset
¶
ImputationPandasDataset(data: dict[str, DataFrame], split: str = train, vars: dict[str, Union[str, list[str]]] | object = REQUIRED, mask_proportion=0.3, mask_method='MCAR', mask_observation_proportion=0.3, ram_cache: bool = True, name: str = '')
Bases: CommonPandasDataset
Subclass of Common Dataset that contains data for imputation models.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
Dict[str, DataFrame]
|
data to use |
required |
split
|
str
|
split to apply. Defaults to Split.train. |
train
|
vars
|
Dict[str, str]
|
contains names of columns in the data. Defaults to gin.REQUIRED. |
REQUIRED
|
mask_proportion
|
float
|
proportion to artificially mask for amputation. Defaults to 0.3. |
0.3
|
mask_method
|
str
|
masking mechanism. Defaults to "MCAR". |
'MCAR'
|
mask_observation_proportion
|
float
|
poportion of the observed data to be masked. Defaults to 0.3. |
0.3
|
ram_cache
|
bool
|
if the dataset should be completely stored in ram and not generated on the fly during training. Defaults to True. |
True
|
name
|
str
|
name of the dataset, used in log messages. Defaults to "". |
''
|
Source code in icu_benchmarks/data/loader.py
ImputationPredictionDataset
¶
ImputationPredictionDataset(data: DataFrame, grouping_column: Optional[str] = 'stay_id', select_columns: Optional[list[str]] = None, ram_cache: bool = True)
Bases: Dataset
Subclass of torch dataset that represents data with missingness for imputation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data
|
DataFrame
|
dict of the different splits of the data |
required |
grouping_column
|
str
|
column that is used for grouping. Defaults to "stay_id". |
'stay_id'
|
select_columns
|
List[str]
|
the columns to serve as input for the imputation model. Defaults to None. |
None
|
ram_cache
|
bool
|
whether the dataset should be stored in ram. Defaults to True. |
True
|
Source code in icu_benchmarks/data/loader.py
check_class_labels
¶
Balanced weights are indexed by class, so the labels must be 0..K-1.
Source code in icu_benchmarks/data/loader.py
icu_benchmarks.data.target_transform
¶
Single-target affine normalization shared by preprocessing, ML and DL.
Classes:
| Name | Description |
|---|---|
TargetTransform |
z = y * scale + offset. Scalar arithmetic preserves NumPy/Torch shape and device. |
Functions:
| Name | Description |
|---|---|
target_scaling_policy |
Resolve legacy bounds without silently accepting partial or conflicting settings. |
transform_outcomes |
Fit once before padding; reuse on all splits (or use a saved source transform). |
TargetTransform
dataclass
¶
TargetTransform(label: str, policy: str, scale: float, offset: float, minimum: float, maximum: float, version: int = 1, metric_space: str = 'original_target_units')
z = y * scale + offset. Scalar arithmetic preserves NumPy/Torch shape and device.
target_scaling_policy
¶
Resolve legacy bounds without silently accepting partial or conflicting settings.
Source code in icu_benchmarks/data/target_transform.py
transform_outcomes
¶
Fit once before padding; reuse on all splits (or use a saved source transform).
Source code in icu_benchmarks/data/target_transform.py
icu_benchmarks.data.pooling
¶
Classes:
| Name | Description |
|---|---|
PooledData |
|
PooledData
¶
PooledData(data_dir, vars, datasets, file_names, shuffle=True, stratify=None, runmode=classification, save_test=True)
Generate pooled data from existing datasets. Args: data_dir: Where to read the data from vars: Variables dictionary datasets: Which datasets to pool file_names: Which files to read from shuffle: Whether to shuffle data; classification pools are stratified, which needs shuffling stratify: Stratify data runmode: Which task runmode save_test: Save left over test data to test on without leakage
Methods:
| Name | Description |
|---|---|
generate |
Generate pooled data from existing datasets. |
Source code in icu_benchmarks/data/pooling.py
generate
¶
Generate pooled data from existing datasets. Args: datasets: Which datasets to pool samples: Amount of samples to pool seed: Random seed