Entry point and run utilities¶
The icu-benchmarks command is icu_benchmarks.run:main. It parses the arguments, binds the gin configs and runs repeated cross-validation through execute_repeated_cv.
icu_benchmarks.run
¶
Main module.
icu_benchmarks.run_utils
¶
Functions:
| Name | Description |
|---|---|
build_parser |
Builds an ArgumentParser for the command line. |
parse_args |
Parses the command line, rejecting unknown arguments; the evaluate command is the same as --eval. |
create_run_dir |
Creates a log directory with the current time as name. |
load_custom_preprocessor |
Imports the CustomPreprocessor class of a file, once per file: gin registers its configurables on import. |
custom_preprocessor |
Instantiates the CustomPreprocessor class of a file. |
import_preprocessor |
Imports a custom preprocessor file before the configs are parsed, so that they can bind its gin parameters. |
bind_custom_preprocessor |
Selects the custom preprocessor. Call this after parsing the configs, which bind the default one (#274). |
aggregate_results |
Aggregates results from all folds and writes to JSON file. |
name_datasets |
Names the datasets for logging (optional). |
log_full_line |
Logs a full line of a given character with a message centered. |
load_pretrained_imputation_model |
Loads a pretrained imputation model. |
bind_selected_label |
Bind the label that preprocessing trains on into the vars macro, so the datasets use that label too. |
check_explain_features |
Fail before tuning, rather than after it, when explanations are requested for a model that cannot provide them. |
setup_logging |
Set up all loggers to use the same format and date format. |
get_config_files |
Get all task and model config files in the specified directory. |
build_parser
¶
build_parser() -> ArgumentParser
Builds an ArgumentParser for the command line.
Returns:
| Type | Description |
|---|---|
ArgumentParser
|
The configured ArgumentParser. |
Source code in icu_benchmarks/run_utils.py
25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 | |
parse_args
¶
Parses the command line, rejecting unknown arguments; the evaluate command is the same as --eval.
Source code in icu_benchmarks/run_utils.py
create_run_dir
¶
Creates a log directory with the current time as name.
Also creates a file in the log directory, if any parameters were randomly searched. The filename contains the fixed hyperparameters to check against in future runs.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
log_dir
|
Path
|
Parent directory to create run directory in. |
required |
randomly_searched_params
|
str
|
String representing the randomly searched params. |
None
|
Returns:
| Type | Description |
|---|---|
Path
|
Path to the created run log directory. |
Source code in icu_benchmarks/run_utils.py
load_custom_preprocessor
cached
¶
Imports the CustomPreprocessor class of a file, once per file: gin registers its configurables on import.
Source code in icu_benchmarks/run_utils.py
custom_preprocessor
¶
custom_preprocessor(path: str = REQUIRED, **kwargs)
Instantiates the CustomPreprocessor class of a file.
preprocess.preprocessor refers to this function instead of the class: gin cannot save a class it does not know, so train_config.gin records the file, and the runs that parse it (--eval, fine-tuning, --samples) load the same class.
Source code in icu_benchmarks/run_utils.py
import_preprocessor
¶
Imports a custom preprocessor file before the configs are parsed, so that they can bind its gin parameters.
bind_custom_preprocessor
¶
Selects the custom preprocessor. Call this after parsing the configs, which bind the default one (#274).
Source code in icu_benchmarks/run_utils.py
aggregate_results
¶
Aggregates results from all folds and writes to JSON file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
log_dir
|
Path
|
Path to the log directory. |
required |
execution_time
|
timedelta
|
Overall execution time. |
None
|
Source code in icu_benchmarks/run_utils.py
221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 | |
name_datasets
¶
Names the datasets for logging (optional).
log_full_line
¶
Logs a full line of a given character with a message centered.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
msg
|
str
|
Message to log. |
required |
level
|
int
|
Logging level. |
INFO
|
char
|
str
|
Character to use for the line. |
'-'
|
num_newlines
|
int
|
Number of newlines to append. |
0
|
Source code in icu_benchmarks/run_utils.py
load_pretrained_imputation_model
¶
Loads a pretrained imputation model.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
use_pretrained_imputation
|
Path to the pretrained imputation model. |
required |
Source code in icu_benchmarks/run_utils.py
bind_selected_label
¶
bind_selected_label(label: str | None = None) -> None
Bind the label that preprocessing trains on into the vars macro, so the datasets use that label too.
preprocess_data picks one of the configured labels (--label, else a preprocess.label binding, else the first), but the datasets get their own copy of %vars from gin and would otherwise read every configured label.
Source code in icu_benchmarks/run_utils.py
check_explain_features
¶
check_explain_features(explain_features: bool | None) -> None
Fail before tuning, rather than after it, when explanations are requested for a model that cannot provide them.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
explain_features
|
bool | None
|
The --explain-features flag; None falls back to the train_common.explain_features binding. |
required |
Source code in icu_benchmarks/run_utils.py
setup_logging
¶
Set up all loggers to use the same format and date format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
date_format
|
Format for the date. |
required | |
log_format
|
Format for the log. |
required | |
verbose
|
Whether to log debug messages. |
required |
Source code in icu_benchmarks/run_utils.py
get_config_files
¶
get_config_files(config_dir: Path)
Get all task and model config files in the specified directory. Args: config_dir: Name of the directory containing the config gin files.
Returns:
| Name | Type | Description |
|---|---|---|
tasks |
List of task names |
|
models |
List of model names |
Source code in icu_benchmarks/run_utils.py
icu_benchmarks.cross_validation
¶
Functions:
| Name | Description |
|---|---|
execute_repeated_cv |
Preprocesses data and trains a model for each fold. |
execute_repeated_cv
¶
execute_repeated_cv(data_dir: Path, log_dir: Path, seed: int, eval_only: bool = False, train_size: Optional[int] = None, load_weights: bool = False, source_dir: Path = Path(''), cv_repetitions: int = 5, cv_repetitions_to_train: Optional[int] = None, cv_folds: int = 5, cv_folds_to_train: Optional[int] = None, reproducible: bool = True, debug: bool = False, generate_cache: bool = False, load_cache: bool = False, test_on: str = 'test', mode: RunMode = classification, pretrained_imputation_model: Optional[str] = None, cpu: bool = False, verbose: bool = False, wandb: bool = False, complete_train: bool = False, explain_features: Optional[bool] = None) -> float
Preprocesses data and trains a model for each fold.
Args:
complete_train: Use the full data for training instead of held out test splits.
explain_features: Save SHAP values of the test set for each fold; None leaves the train_common binding in effect.
wandb: Use wandb for logging.
data_dir: Path to the data directory.
log_dir: Path to the log directory.
seed: Random seed.
eval_only: Whether to only evaluate the model.
train_size: Fixed size of train split (including validation data).
load_weights: Whether to load weights from source_dir.
source_dir: Path to the source directory.
cv_folds: Number of folds for cross validation.
cv_folds_to_train: Number of folds to use during training. If None, all folds are trained on.
cv_repetitions: Amount of cross validation repetitions.
cv_repetitions_to_train: Amount of training repetitions. If None, all repetitions are trained on.
reproducible: Whether to make torch reproducible.
debug: Whether to load less data and enable more logging.
generate_cache: Whether to generate and save cache.
load_cache: Whether to load previously cached data.
test_on: Dataset to test on. Can be "test" or "val" (e.g. for hyperparameter tuning).
mode: Run mode. Can be one of the values of RunMode
pretrained_imputation_model: Use a pretrained imputation model.
cpu: Whether to run on CPU.
verbose: Enable detailed logging.
Returns: The average native-unit MSE for regression, or loss for other run modes.
Source code in icu_benchmarks/cross_validation.py
19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 | |
icu_benchmarks.constants
¶
icu_benchmarks.wandb_utils
¶
Functions:
| Name | Description |
|---|---|
wandb_running |
Check if wandb is running. |
update_wandb_config |
updates wandb config if wandb is running |
apply_wandb_sweep |
applies the wandb sweep configuration to the namespace object |
wandb_log |
logs metrics to wandb |
set_wandb_experiment_name |
stores the run name in wandb config |
update_wandb_config
¶
update_wandb_config(config: dict) -> None
updates wandb config if wandb is running
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
dict
|
config to set |
required |
Source code in icu_benchmarks/wandb_utils.py
apply_wandb_sweep
¶
applies the wandb sweep configuration to the namespace object
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args
|
Namespace
|
parsed arguments |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Namespace |
Namespace
|
arguments with sweep configuration applied (some are applied via hyperparams) |
Source code in icu_benchmarks/wandb_utils.py
wandb_log
¶
logs metrics to wandb
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
log_dict
|
dict
|
metric dict to log |
required |
set_wandb_experiment_name
¶
stores the run name in wandb config
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
args
|
Namespace
|
parsed arguments |
required |
mode
|
RunMode
|
run mode |
required |