Preprocessing¶
Preprocessing turns the raw cohort into model inputs. It runs once per fold, after the stays are split, and is fitted on the training split only, so no information from the validation or test stays leaks into it. The steps are ReciPys recipes.
The default pipelines¶
The task configs select a preprocessor with preprocess.preprocessor:
| Gin name | Class | Used by |
|---|---|---|
base_classification_preprocessor |
PolarsClassificationPreprocessor |
BinaryClassification.gin |
base_regression_preprocessor |
PolarsRegressionPreprocessor |
Regression.gin (adds target scaling) |
base_imputation_preprocessor |
PandasImputationPreprocessor |
DatasetImputation.gin |
The classification and regression pipelines apply these steps:
Static variables
- Add a missingness indicator for every variable.
- Scale the numeric variables (with
scaling=True) and fill their missing values with 0, the training mean. - Fill missing categorical values with the most frequent one and encode the categories as integers (unseen categories become -1).
Dynamic variables
- Scale the variables (with
scaling=True). - Add a
MissingIndicator_<name>column that is 1 where a value was not measured. - With
generate_features=True(the default for classical models), add features over the stay so far: the count of measurements, and the minimum, maximum and mean of each variable. - With a pretrained imputation model (
-imp), impute the missing values with it. - Fill the remaining missing values forward in time, then with 0 (the training mean) before the first measurement.
Arguments of the preprocessors, settable with gin:
| Argument | Default | Meaning |
|---|---|---|
generate_features |
True for ML, False for DL (set in MLCommon.gin and DLCommon.gin) |
Add the summary features over time. |
scaling |
True |
Scale the features. |
use_static_features |
True |
Use the static variables. |
vars_to_exclude |
None |
Variables that get no missingness indicators or generated features. |
For example, -hp 'base_classification_preprocessor.scaling=False' turns off feature scaling.
Writing a custom preprocessor¶
A preprocessor subclasses Preprocessor and implements two methods:
apply(data, vars)takes the split data,data[split][segment]for thetrain,valandtestsplits and theSTATIC,DYNAMICandOUTCOMEsegments, and returns it preprocessed, with the features in aFEATURESsegment.to_cache_string()returns a string that identifies the preprocessor and its settings; it is part of the cache key.
The simplest way is to subclass a default preprocessor and change one part:
import gin
from icu_benchmarks.data.preprocessor import PolarsClassificationPreprocessor
@gin.configurable("my_preprocessor")
class CustomPreprocessor(PolarsClassificationPreprocessor):
def __init__(self, *args, clip_quantile: float = 0.99, **kwargs):
super().__init__(*args, **kwargs)
self.clip_quantile = clip_quantile
def _process_dynamic(self, data, vars):
# change the dynamic steps here
return super()._process_dynamic(data, vars)
def to_cache_string(self):
return super().to_cache_string() + f"_clip_{self.clip_quantile}"
Pass the file with -p:
- The file must define a class named
CustomPreprocessor; YAIB raises an error otherwise. - YAIB imports the file before it parses the configs, so the configs and
-hpcan bind the preprocessor's gin parameters, and it selects the preprocessor after the configs, so the task config's default does not replace it. - The run's
train_config.ginrecords the file path. Evaluation, fine-tuning and--samplesruns that read that config load the same preprocessor; keep the file at its path.
To make a preprocessor the default for a task instead, register it with gin and bind it in the task config:
preprocess.preprocessor = @my_preprocessor.