Feature explanations¶
With --explain-features, YAIB explains the test predictions of each fold, to show which features drive a model's
predictions:
- Tree-based models get SHAP values.
- Deep learning models get attributions computed with Captum, for every labeled hour, together with metrics of how faithful these attributions are.
Explanations are not computed during tuning trials, which would be slow and unused. YAIB checks before tuning starts that the model supports them, so an unsupported model fails immediately instead of after a long run.
Tree-based models: SHAP values¶
uv run icu-benchmarks train \
-d demo_data/mortality24/mimic_demo -n mimic_demo \
-t BinaryClassification -tn Mortality24 \
-m LGBMClassifier \
-l ../yaib_logs/ \
--explain-features
SHAP values are computed with shap.TreeExplainer, so they are available for LGBMClassifier, LGBMRegressor,
XGBClassifier, CBClassifier, RFClassifier and BRFClassifier: the models whose class sets explainer_cls.
Linear models, SVMs and RUSBClassifier do not have an explainer yet.
Each fold writes test_shap_values.parquet: one row per test sample and one column per model input (the
trained_columns, including generated features). For regression models, the values are in native target units and the
base values are saved as test_shap_base_values.npy. After all folds, YAIB concatenates the folds' values into
aggregated_shap_values.parquet in the run folder.
Deep learning models: attributions and their quality¶
Install the optional Captum dependency first:
uv run icu-benchmarks train \
-d demo_data/aki/eicu_demo -n eicu_demo \
-t BinaryClassification -tn AKI \
-m GRU \
-l ../yaib_logs/ \
--explain-features
This works for all deep learning prediction models (GRU, LSTM, RNN, GRUD, TCN, Transformer,
LocalTransformer, BAT, Mamba) on binary classification and regression tasks, with the default Polars datasets
(train_common.polars = True).
What is explained¶
- Every labeled hour. For hourly tasks such as AKI and sepsis, each hour with a label is explained separately; for
once-per-stay tasks such as mortality, the one labeled hour. A sample of
max_staystest stays (100 by default) is explained, to bound the cost. - The model output at that hour: the log-odds of the positive class for binary classification, the predicted value for regression (attributions are converted to native target units).
- Variables, not columns. A variable is a value column together with its
MissingIndicator_<name>column. Its attribution is summed over both columns and over all hours up to the explained one. Removing a variable means "not measured": its value is set to 0 (the training mean, after standardisation) and its missingness indicator to 1, for its whole history. - Columns without a missingness indicator are set to 0 as well, so for them removal means "replaced by the training
mean" (or by 0 without scaling), not "not measured". These are the variables in
vars_to_excludeand, when feature generation is on, the generated features (counterand the*_histcolumns); deep learning models do not generate features by default.
Methods¶
| Method | What it computes |
|---|---|
IntegratedGradients |
Integrated Gradients from the "not measured" input to the actual input. The attributions sum to the output difference; the mean deviation is reported as convergence_delta. |
FeatureAblation |
The output drop when a variable is removed (occlusion of its whole history). |
Random |
Random attributions: the reference that the other methods should beat on every metric. |
Quality metrics¶
| Metric | Meaning | Better |
|---|---|---|
faithfulness |
Correlation, over random variable subsets, between a subset's total attribution and the output drop when it is removed (Bhatt et al., 2020). The same subsets are used for every method. | Higher (Random ≈ 0) |
deletion |
Mean output drop when the 1, 2, ..., 10 highest-attributed variables are removed (area over the perturbation curve). | Higher |
sparsity |
Gini index of the absolute attributions: whether a few variables carry the explanation. | Context-dependent |
Faithfulness and deletion tell you whether the attributions reflect what the model does, not whether the model is right:
judge them against the Random reference and together with the model's test performance.
Output¶
Each fold writes:
| File | Contents |
|---|---|
test_attributions.parquet |
One row per method and explained hour: the stay ID, the hour, the method and one column per variable. |
test_feature_importance.parquet |
Per method and variable, the mean absolute and the mean attribution. |
test_explanation_metrics.json |
The metrics, keyed explain/<method>/<metric>, averaged over the explained hours. |
The metrics are averaged over the folds into accumulated_test_metrics.json like the other test metrics, and the
feature importances of all folds are collected in aggregated_feature_importance.parquet.
Settings¶
The explain_predictions configurable controls the methods, metrics and cost:
| Binding | Default | Meaning |
|---|---|---|
explain_predictions.methods |
("IntegratedGradients", "FeatureAblation", "Random") |
Attribution methods |
explain_predictions.metrics |
("faithfulness", "deletion", "sparsity") |
Quality metrics |
explain_predictions.max_stays |
100 |
Test stays to explain; every labeled hour of each is explained |
explain_predictions.ig_steps |
32 |
Integration steps of Integrated Gradients |
explain_predictions.faithfulness_subsets |
20 |
Random variable subsets per stay |
explain_predictions.faithfulness_subset_size |
0.1 |
Share of the variables in each subset |
explain_predictions.deletion_steps |
10 |
Top variables removed for the deletion metric |
explain_predictions.batch_size |
64 |
Inputs per forward pass |
The cost grows with the number of explained hours, so hourly tasks with long stays are the most expensive: on the
AKI demo cohort, 30 stays (about 1,300 labeled hours) take about 10 seconds on a laptop CPU. Lower max_stays or
ig_steps for large cohorts, for example -hp explain_predictions.max_stays=50.
Enabling it in a config¶
Instead of the flag, you can bind train_common.explain_features = True in a model or experiment config. The flag
overrides the binding: --explain-features turns explanations on and --no-explain-features turns them off.