Skip to content

Deep learning models

PyTorch Lightning prediction models. Each is a gin configurable named in its configs/prediction_models/*.gin file.

icu_benchmarks.models.dl_models.rnn

Classes:

Name Description
RNNet

Torch standard RNN model

LSTMNet

Torch standard LSTM model.

GRUNet

Torch standard GRU model.

RNNet

RNNet(input_size, hidden_dim, layer_dim, num_classes, *args, **kwargs)

Bases: DLPredictionWrapper

Torch standard RNN model

Source code in icu_benchmarks/models/dl_models/rnn.py
def __init__(self, input_size, hidden_dim, layer_dim, num_classes, *args, **kwargs):
    super().__init__(
        *args,
        input_size=input_size,
        hidden_dim=hidden_dim,
        layer_dim=layer_dim,
        num_classes=num_classes,
        **kwargs,
    )
    self.hidden_dim = hidden_dim
    self.layer_dim = layer_dim
    self.rnn = nn.RNN(input_size[2], hidden_dim, layer_dim, batch_first=True)
    self.logit = nn.Linear(hidden_dim, num_classes)

LSTMNet

LSTMNet(input_size, hidden_dim, layer_dim, num_classes, *args, **kwargs)

Bases: DLPredictionWrapper

Torch standard LSTM model.

Source code in icu_benchmarks/models/dl_models/rnn.py
def __init__(self, input_size, hidden_dim, layer_dim, num_classes, *args, **kwargs):
    super().__init__(
        *args,
        input_size=input_size,
        hidden_dim=hidden_dim,
        layer_dim=layer_dim,
        num_classes=num_classes,
        **kwargs,
    )
    self.hidden_dim = hidden_dim
    self.layer_dim = layer_dim
    self.rnn = nn.LSTM(input_size[2], hidden_dim, layer_dim, batch_first=True)
    self.logit = nn.Linear(hidden_dim, num_classes)

GRUNet

GRUNet(input_size, hidden_dim, layer_dim, num_classes, *args, **kwargs)

Bases: DLPredictionWrapper

Torch standard GRU model.

Source code in icu_benchmarks/models/dl_models/rnn.py
def __init__(self, input_size, hidden_dim, layer_dim, num_classes, *args, **kwargs):
    super().__init__(
        *args,
        input_size=input_size,
        hidden_dim=hidden_dim,
        layer_dim=layer_dim,
        num_classes=num_classes,
        **kwargs,
    )
    self.hidden_dim = hidden_dim
    self.layer_dim = layer_dim
    self.rnn = nn.GRU(input_size[2], hidden_dim, layer_dim, batch_first=True)
    self.logit = nn.Linear(hidden_dim, num_classes)

icu_benchmarks.models.dl_models.grud

GRU-D: a GRU that decays inputs and hidden state with the time since each variable was last observed.

GRU-D (Che et al., Recurrent Neural Networks for Multivariate Time Series with Missing Values, Scientific Reports 8, 6085, 2018, https://doi.org/10.1038/s41598-018-24271-9) feeds a GRU the observation mask m_t and decays both the imputed inputs and the hidden state with the time delta_t since each variable was last observed (equations 2 and 10 to 16 of the paper):

gamma_x = exp(-max(0, W_gamma_x delta_t + b_gamma_x))      (W_gamma_x diagonal)
gamma_h = exp(-max(0, W_gamma_h delta_t + b_gamma_h))
x_hat_t = m_t x_t + (1 - m_t) (gamma_x x_last + (1 - gamma_x) x_mean)
h_hat   = gamma_h h_{t-1}
z_t, r_t = sigmoid(W x_hat_t + U h_hat + V m_t + b)        (one set of weights per gate)
h_t     = (1 - z_t) h_hat + z_t tanh(W x_hat_t + U (r_t h_hat) + V m_t + b)

Ported from the YAIB fork by Katja Jagd and Rachael DeVries (https://github.com/Katja-Jagd/YAIB, branch katja, icu_benchmarks/models/dl_models/grud.py, MIT License), which was used for BAT pretraining on YAIB (Jagd et al., https://arxiv.org/abs/2509.19885) and as a baseline for the Bi-Axial Transformer (DeVries et al., https://arxiv.org/abs/2508.12418). The fork's cell is a PyTorch translation of the TensorFlow GRU-D in SeFT (Horn et al., https://github.com/BorgwardtLab/Set_Functions_for_Time_Series, seft/models/gru_d.py, BSD 3-Clause License), which adapts the Keras implementation by the first author of GRU-D (https://github.com/PeterChe1990/GRU-D, nn_utils/grud_layers.py, MIT License). Both license notices are reproduced at the end of this docstring.

How the port fits YAIB:

  • Inputs. The model takes the same hourly feature matrix as YAIB's GRU. The observation mask m_t comes from the MissingIndicator_<name> columns of the preprocessing recipe (see ObservationMaskWrapper); columns without an indicator, including the indicators themselves, count as always observed. YAIB's data is on a regular hourly grid, so delta_t counts hours since the last measurement. The features are standardised, so the empirical mean x_mean is 0, which is also the value the recipe fills in before the first measurement. Static features are inputs at every hour, as for the other YAIB models, instead of initialising the hidden state as in the fork and SeFT.
  • Output. The model predicts at every hour from the hidden state h_t, so it is causal by construction. Per-stay tasks use the prediction at the last observed hour, which is the fork's pooling="hidden".
  • Fixes to the fork. The decay parameters were initialised to exactly zero, where relu has no gradient, so the decays never trained; their biases now start slightly positive (decay rates close to 1, as before). The gate biases were never initialised and were not used when the mask was fed to the gates (the Keras reference always adds them). Recurrent and mask dropout were never applied because of an inverted condition. The new state interpolates with the decayed state h_hat, as in equation 16 of the paper (the Keras reference used the undecayed h_{t-1}). The cell is a plain module instead of TorchScript, so the model can be pickled with save_model.
  • Left out. The fork's options for ablations and other tasks: zero, forward or raw imputation instead of the input decay, masking decay, indicator-only or value-only inputs, mean or max pooling over time, the static encoder, self-supervised pretraining (SSL_GRUD) and GRU-D as an imputation model.

License notices of the upstream implementations:

SeFT (https://github.com/BorgwardtLab/Set_Functions_for_Time_Series), BSD 3-Clause License:

Copyright 2020 Max Horn

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its
   contributors may be used to endorse or promote products derived from
   this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AD CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

GRU-D Keras implementation (https://github.com/PeterChe1990/GRU-D), MIT License:

Copyright (c) 2018 Zhengping Che

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of
the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO
THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Classes:

Name Description
GRUDLayer

GRU-D recurrence over a whole sequence.

GRUD

GRU-D (Che et al., 2018) with one prediction per hour; see the module docstring.

Functions:

Name Description
decay_rate

Decay rate exp(-max(0, x)) in (0, 1], monotonically decreasing in the time since the last observation.

variational_dropout

One dropout mask per sequence, shared by all time steps (Gal and Ghahramani, 2016), as in the Keras GRU-D.

GRUDLayer

GRUDLayer(input_dim: int, hidden_dim: int, dropout: float = 0.0, recurrent_dropout: float = 0.0)

Bases: Module

GRU-D recurrence over a whole sequence.

Parameters:

Name Type Description Default
input_dim int

Number of input columns.

required
hidden_dim int

Size of the hidden state.

required
dropout float

Dropout on the inputs and the mask, with one mask per sequence.

0.0
recurrent_dropout float

Dropout on the hidden state in the recurrent connections, with one mask per sequence.

0.0

Methods:

Name Description
reset_parameters

Initialise as the Keras reference: Glorot-uniform kernels, orthogonal recurrent kernels, zero gate biases.

forward

Run GRU-D over the sequence.

Source code in icu_benchmarks/models/dl_models/grud.py
def __init__(self, input_dim: int, hidden_dim: int, dropout: float = 0.0, recurrent_dropout: float = 0.0):
    super().__init__()
    self.hidden_dim = hidden_dim
    self.dropout = dropout
    self.recurrent_dropout = recurrent_dropout
    # Input decay: one rate per column (diagonal W_gamma_x)
    self.input_decay_weight = nn.Parameter(torch.empty(input_dim))
    self.input_decay_bias = nn.Parameter(torch.empty(input_dim))
    # Hidden decay: a full matrix W_gamma_h
    self.hidden_decay = nn.Linear(input_dim, hidden_dim)
    # W x_hat + b and V m for the update gate, the reset gate and the candidate state, in this order
    self.input_gates = nn.Linear(input_dim, 3 * hidden_dim)
    self.mask_gates = nn.Linear(input_dim, 3 * hidden_dim, bias=False)
    # U h_hat for the update and reset gates, and U (r * h_hat) for the candidate state
    self.recurrent_gates = nn.Linear(hidden_dim, 2 * hidden_dim, bias=False)
    self.recurrent_candidate = nn.Linear(hidden_dim, hidden_dim, bias=False)
    self.reset_parameters()

reset_parameters

reset_parameters()

Initialise as the Keras reference: Glorot-uniform kernels, orthogonal recurrent kernels, zero gate biases.

The decays start without decay, as in the reference, but with a small positive bias instead of zero: relu has no gradient at zero, so all-zero decay parameters would never train.

Source code in icu_benchmarks/models/dl_models/grud.py
def reset_parameters(self):
    """Initialise as the Keras reference: Glorot-uniform kernels, orthogonal recurrent kernels, zero gate biases.

    The decays start without decay, as in the reference, but with a small positive bias instead of zero: relu has no
    gradient at zero, so all-zero decay parameters would never train.
    """
    for gates in (self.input_gates, self.mask_gates):
        for kernel in gates.weight.split(self.hidden_dim):
            nn.init.xavier_uniform_(kernel)
    for kernel in (*self.recurrent_gates.weight.split(self.hidden_dim), self.recurrent_candidate.weight):
        nn.init.orthogonal_(kernel)
    nn.init.zeros_(self.input_gates.bias)
    for weight, bias in ((self.input_decay_weight, self.input_decay_bias), self.hidden_decay.parameters()):
        nn.init.zeros_(weight)
        nn.init.constant_(bias, DECAY_BIAS_INIT)

forward

forward(x: Tensor, observed: Tensor) -> Tensor

Run GRU-D over the sequence.

Parameters:

Name Type Description Default
x Tensor

Inputs of shape (batch, time, columns). Values that were not observed are ignored, except as a fallback for the first hours before a column was measured, where the empirical mean 0 is used.

required
observed Tensor

Observation mask of the same shape, 1 where x was measured and 0 where it was imputed.

required

Returns:

Type Description
Tensor

Hidden states of shape (batch, time, hidden_dim).

Source code in icu_benchmarks/models/dl_models/grud.py
def forward(self, x: torch.Tensor, observed: torch.Tensor) -> torch.Tensor:
    """Run GRU-D over the sequence.

    Args:
        x: Inputs of shape (batch, time, columns). Values that were not observed are ignored, except as a fallback
            for the first hours before a column was measured, where the empirical mean 0 is used.
        observed: Observation mask of the same shape, 1 where x was measured and 0 where it was imputed.

    Returns:
        Hidden states of shape (batch, time, hidden_dim).
    """
    batch, steps, _ = x.shape
    # delta_t: hours since each column was last observed before t (equation 2), and that last observed value
    deltas, last_values = [], []
    delta, last = torch.zeros_like(x[:, 0]), torch.zeros_like(x[:, 0])
    for t in range(steps):
        deltas.append(delta)
        last_values.append(last)
        delta = 1 + (1 - observed[:, t]) * delta
        last = observed[:, t] * x[:, t] + (1 - observed[:, t]) * last
    delta, last = torch.stack(deltas, dim=1), torch.stack(last_values, dim=1)

    # Everything except the recurrence depends on the inputs only, so it is computed for all time steps at once
    gamma_x = decay_rate(delta * self.input_decay_weight + self.input_decay_bias)
    x_hat = observed * x + (1 - observed) * gamma_x * last
    gamma_h = decay_rate(self.hidden_decay(delta))
    x_hat = variational_dropout(x_hat, self.dropout, self.training)
    mask = variational_dropout(observed, self.dropout, self.training)
    gate_inputs = self.input_gates(x_hat) + self.mask_gates(mask)
    gate_inputs, candidate_inputs = gate_inputs.split([2 * self.hidden_dim, self.hidden_dim], dim=-1)

    h = x.new_zeros(batch, self.hidden_dim)
    keep = variational_dropout(x.new_ones(batch, self.hidden_dim), self.recurrent_dropout, self.training)
    outputs = []
    for t in range(steps):
        h = gamma_h[:, t] * h
        update, reset = torch.sigmoid(gate_inputs[:, t] + self.recurrent_gates(h * keep)).chunk(2, dim=-1)
        candidate = torch.tanh(candidate_inputs[:, t] + self.recurrent_candidate(reset * h * keep))
        h = (1 - update) * h + update * candidate
        outputs.append(h)
    return torch.stack(outputs, dim=1)

GRUD

GRUD(input_size, hidden_dim, num_classes, dropout=0.0, recurrent_dropout=0.0, *args, **kwargs)

Bases: ObservationMaskWrapper

GRU-D (Che et al., 2018) with one prediction per hour; see the module docstring.

Parameters:

Name Type Description Default
input_size

Shape (batch, time, columns) of the input.

required
hidden_dim

Size of the hidden state.

required
num_classes

Number of classes, or 1 for regression.

required
dropout

Dropout on the inputs and the observation mask, with one mask per sequence.

0.0
recurrent_dropout

Dropout on the hidden state in the recurrent connections, with one mask per sequence.

0.0
Source code in icu_benchmarks/models/dl_models/grud.py
def __init__(self, input_size, hidden_dim, num_classes, dropout=0.0, recurrent_dropout=0.0, *args, **kwargs):
    super().__init__(
        *args,
        input_size=input_size,
        hidden_dim=hidden_dim,
        num_classes=num_classes,
        dropout=dropout,
        recurrent_dropout=recurrent_dropout,
        **kwargs,
    )
    self.grud = GRUDLayer(input_size[2], hidden_dim, dropout=dropout, recurrent_dropout=recurrent_dropout)
    self.logit = nn.Linear(hidden_dim, num_classes)

decay_rate

decay_rate(pre_activation: Tensor) -> Tensor

Decay rate exp(-max(0, x)) in (0, 1], monotonically decreasing in the time since the last observation.

Source code in icu_benchmarks/models/dl_models/grud.py
def decay_rate(pre_activation: torch.Tensor) -> torch.Tensor:
    """Decay rate exp(-max(0, x)) in (0, 1], monotonically decreasing in the time since the last observation."""
    return torch.exp(-torch.relu(pre_activation))

variational_dropout

variational_dropout(x: Tensor, p: float, training: bool) -> Tensor

One dropout mask per sequence, shared by all time steps (Gal and Ghahramani, 2016), as in the Keras GRU-D.

Source code in icu_benchmarks/models/dl_models/grud.py
def variational_dropout(x: torch.Tensor, p: float, training: bool) -> torch.Tensor:
    """One dropout mask per sequence, shared by all time steps (Gal and Ghahramani, 2016), as in the Keras GRU-D."""
    if not training or p == 0:
        return x
    shape = (x.shape[0], 1, x.shape[-1]) if x.dim() == 3 else x.shape
    return x * F.dropout(x.new_ones(shape), p=p, training=True)

icu_benchmarks.models.dl_models.tcn

Classes:

Name Description
TemporalConvNet

Temporal Convolutional Network. Adapted from TCN original paper https://github.com/locuslab/TCN

TemporalConvNet

TemporalConvNet(input_size, num_channels, num_classes, *args, max_seq_length=0, kernel_size=2, dropout=0.0, **kwargs)

Bases: DLPredictionWrapper

Temporal Convolutional Network. Adapted from TCN original paper https://github.com/locuslab/TCN

Source code in icu_benchmarks/models/dl_models/tcn.py
def __init__(
    self,
    input_size,
    num_channels,
    num_classes,
    *args,
    max_seq_length=0,
    kernel_size=2,
    dropout=0.0,
    **kwargs,
):
    super().__init__(
        *args,
        input_size=input_size,
        num_channels=num_channels,
        num_classes=num_classes,
        max_seq_length=max_seq_length,
        kernel_size=kernel_size,
        dropout=dropout,
        **kwargs,
    )
    layers = []

    # We compute automatically the depth based on the desired seq_length.
    if isinstance(num_channels, Integral) and max_seq_length:
        num_channels = [num_channels] * int(np.ceil(np.log(max_seq_length / 2) / np.log(kernel_size)))
    elif isinstance(num_channels, Integral) and not max_seq_length:
        raise Exception("a maximum sequence length needs to be provided if num_channels is int")

    num_levels = len(num_channels)
    for i in range(num_levels):
        dilation_size = 2**i
        in_channels = input_size[2] if i == 0 else num_channels[i - 1]
        out_channels = num_channels[i]
        layers += [
            TemporalBlock(
                in_channels,
                out_channels,
                kernel_size,
                stride=1,
                dilation=dilation_size,
                padding=(kernel_size - 1) * dilation_size,
                dropout=dropout,
            )
        ]

    self.network = nn.Sequential(*layers)
    self.logit = nn.Linear(num_channels[-1], num_classes)

icu_benchmarks.models.dl_models.transformer

Classes:

Name Description
BaseTransformer
Transformer

Transformer model.

LocalTransformer

Transformer model with local context.

BaseTransformer

BaseTransformer(block_class, input_size, hidden, heads, ff_hidden_mult, depth, num_classes, dropout=0.0, pos_encoding=True, dropout_att=0.0, local_context=None, *args, **kwargs)

Bases: DLPredictionWrapper

Source code in icu_benchmarks/models/dl_models/transformer.py
def __init__(
    self,
    block_class,
    input_size,
    hidden,
    heads,
    ff_hidden_mult,
    depth,
    num_classes,
    dropout=0.0,
    pos_encoding=True,
    dropout_att=0.0,
    local_context=None,
    *args,
    **kwargs,
):
    # Pass input_size on so it is saved in the checkpoint's hyperparameters, which load_from_checkpoint needs (#259)
    super().__init__(*args, input_size=input_size, **kwargs)
    if local_context is not None and self._get_name() == "Transformer":
        raise ValueError("Local context is only supported for LocalTransformer")
    hidden = hidden if hidden % 2 == 0 else hidden + 1  # Make sure hidden is even
    self.input_embedding = nn.Linear(input_size[2], hidden)  # This acts as a time-distributed layer by defaults
    if pos_encoding:
        self.pos_encoder = PositionalEncoding(hidden)
    else:
        self.pos_encoder = None

    t_blocks = []
    for _ in range(depth):
        t_blocks.append(
            block_class(
                emb=hidden,
                hidden=hidden,
                heads=heads,
                mask=True,
                ff_hidden_mult=ff_hidden_mult,
                dropout=dropout,
                dropout_att=dropout_att,
                **({"local_context": local_context} if local_context is not None else {}),
            )
        )

    self.t_blocks = nn.Sequential(*t_blocks)
    self.logit = nn.Linear(hidden, num_classes)

Transformer

Transformer(*args, **kwargs)

Bases: BaseTransformer

Transformer model.

Source code in icu_benchmarks/models/dl_models/transformer.py
def __init__(self, *args, **kwargs):
    super().__init__(TransformerBlock, *args, **kwargs)

LocalTransformer

LocalTransformer(*args, **kwargs)

Bases: BaseTransformer

Transformer model with local context.

Source code in icu_benchmarks/models/dl_models/transformer.py
def __init__(self, *args, **kwargs):
    super().__init__(LocalBlock, *args, **kwargs)

icu_benchmarks.models.dl_models.bat

Bi-Axial Transformer (BAT): attention along both the variable axis and the time axis of the hourly grid.

BAT (DeVries et al., Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification, 2025, https://arxiv.org/abs/2508.12418) turns every (variable, hour) cell of a stay into a token made of the value, whether it was measured, a learned embedding of the variable and an encoding of the time. Two streams of these tokens attend along both axes in opposite orders: one attends across the variables within each hour and then across the hours of each variable, the other across the hours first. Both streams use the same attention layers for each axis, are pooled over the variables and merged by a nonlinear layer. Jagd et al. (https://arxiv.org/abs/2509.19885) pretrained BAT on YAIB.

Ported from the YAIB fork by Katja Jagd and Rachael DeVries (https://github.com/Katja-Jagd/YAIB, branch katja, icu_benchmarks/models/dl_models/bat.py, MIT License). The fork's BAT encoder comes from the BAT authors' reference code, and its causal ("autoregressive") encoder, which this module ports, was added by Rachael DeVries. The time encoding is SeFT's positional encoding (Horn et al., https://github.com/BorgwardtLab/Set_Functions_for_Time_Series, seft/models/transformer.py, BSD 3-Clause License); its license notice is reproduced at the end of this docstring.

How the port fits YAIB:

  • Inputs. The variables ("sensors") are the input columns that are not missingness indicators, static features included. Each token gets the value and the observation flag read from the MissingIndicator_<name> column (see ObservationMaskWrapper); variables without an indicator count as measured. YAIB's data is on a regular hourly grid, so the time of a token is its hour in the stay. Static features are variables like the others instead of a separate embedding that is concatenated after pooling, as in the fork.
  • Output. Attention across hours is causal and every other step works within an hour, so the prediction for hour t only depends on hours up to t. The model predicts at every hour, and per-stay tasks use the prediction at the last observed hour, which has attended to the whole stay. YAIB pads stays at the end, so padding never affects the predictions for real hours and no padding mask is needed.
  • Not ported. The bidirectional encoder of the BAT paper, which max-pools over all variables and hours, is not included: it can only serve per-stay tasks, and YAIB does not pass a padding mask to forward, so it would have to guess the padding from all-zero rows. The fork used it for per-stay tasks and the causal encoder for hourly tasks. Also left out: the pooling and input ablation options, attention-map outputs, the separate prediction heads (the logit layer replaces them), the forecasting head and self-supervised pretraining (SSL_BAT).
  • Attention layers. YAIB's TransformerBlock (post-norm, ReLU feed-forward) replaces the pre-norm blocks of the x-transformers package, which avoids a new dependency. Hyperparameters follow YAIB's Transformer: depth is the fork's layers and dropout_att its attn_dropout.
  • Fixes to the fork. The padding mask of the time attention (use_mask) was repeated in the wrong order, so each row of a batch got the mask of another stay; it is no longer needed. An odd value_embed_size broke the time encoding; it is rounded up to an even size. The placeholder logit layer, which only existed to select the metrics, is now the output layer.

License notice of SeFT (https://github.com/BorgwardtLab/Set_Functions_for_Time_Series), BSD 3-Clause License:

Copyright 2020 Max Horn

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its
   contributors may be used to endorse or promote products derived from
   this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AD CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Classes:

Name Description
BAT

Causal Bi-Axial Transformer with one prediction per hour; see the module docstring.

Functions:

Name Description
time_encoding

SeFT's sinusoidal encoding of the times 0, ..., steps - 1, with timescales from 1 to max_time.

BAT

BAT(input_size, num_classes, value_embed_size=8, depth=1, heads=1, ff_hidden_mult=4, dropout=0.0, dropout_att=0.0, max_time=500, *args, **kwargs)

Bases: ObservationMaskWrapper

Causal Bi-Axial Transformer with one prediction per hour; see the module docstring.

Parameters:

Name Type Description Default
input_size

Shape (batch, time, columns) of the input.

required
num_classes

Number of classes, or 1 for regression.

required
value_embed_size

Size of the embedding of a value and its observation flag (rounded up to an even number). The token size adds an embedding of the variable, of size 1.6 * sqrt(input columns) (the fork used the number of variables, which is only known once the column names are set).

8
depth

Number of attention blocks per axis.

1
heads

Number of attention heads.

1
ff_hidden_mult

Width of the feed-forward layers, as a multiple of the token size.

4
dropout

Dropout in the attention blocks.

0.0
dropout_att

Dropout on the attention outputs.

0.0
max_time

Largest timescale of the time encoding, in hours.

500
Source code in icu_benchmarks/models/dl_models/bat.py
def __init__(
    self,
    input_size,
    num_classes,
    value_embed_size=8,
    depth=1,
    heads=1,
    ff_hidden_mult=4,
    dropout=0.0,
    dropout_att=0.0,
    max_time=500,
    *args,
    **kwargs,
):
    super().__init__(
        *args,
        input_size=input_size,
        num_classes=num_classes,
        value_embed_size=value_embed_size,
        depth=depth,
        heads=heads,
        ff_hidden_mult=ff_hidden_mult,
        dropout=dropout,
        dropout_att=dropout_att,
        max_time=max_time,
        **kwargs,
    )
    num_columns = input_size[2]
    sensor_embed_size = min(600, round(1.6 * num_columns**0.5))
    # The time encoding needs an even token size
    sensor_embed_size += sensor_embed_size % 2
    value_embed_size += value_embed_size % 2
    emb = value_embed_size + sensor_embed_size
    self.max_time = max_time
    self.value_embedding = nn.Linear(2, value_embed_size)
    # One embedding per input column; the rows of the indicator columns stay unused
    self.sensor_embedding = nn.Embedding(num_columns, sensor_embed_size)

    def blocks(causal):
        return nn.Sequential(
            *[
                TransformerBlock(emb, emb, heads, ff_hidden_mult, dropout=dropout, mask=causal, dropout_att=dropout_att)
                for _ in range(depth)
            ]
        )

    # Across the variables within an hour, and causally across the hours of a variable
    self.sensor_attention = blocks(causal=False)
    self.time_attention = blocks(causal=True)
    self.merge = nn.Linear(2 * emb, 2 * emb)
    self.logit = nn.Linear(2 * emb, num_classes)

time_encoding

time_encoding(steps: int, dim: int, max_time: float, device=None) -> Tensor

SeFT's sinusoidal encoding of the times 0, ..., steps - 1, with timescales from 1 to max_time.

Returns:

Type Description
Tensor

Tensor of shape (steps, dim): sines in the first half and cosines in the second half of the last axis.

Source code in icu_benchmarks/models/dl_models/bat.py
def time_encoding(steps: int, dim: int, max_time: float, device=None) -> torch.Tensor:
    """SeFT's sinusoidal encoding of the times 0, ..., steps - 1, with timescales from 1 to max_time.

    Returns:
        Tensor of shape (steps, dim): sines in the first half and cosines in the second half of the last axis.
    """
    timescales = max_time ** torch.linspace(0, 1, dim // 2, device=device)
    scaled = torch.arange(steps, device=device, dtype=torch.float)[:, None] / timescales
    return torch.cat([torch.sin(scaled), torch.cos(scaled)], dim=-1)

icu_benchmarks.models.dl_models.mamba

Mamba selective state-space model for YAIB's hourly time series.

Mamba (Gu and Dao, 2023, https://arxiv.org/abs/2312.00752) replaces attention with a selective state-space layer whose state update depends on the input, so it scales linearly with sequence length. EHRMamba (Fallahpour et al., 2024, https://arxiv.org/abs/2405.14567) showed that this backbone works well for EHR prediction. This module contains the Mamba backbone only, adapted to the inputs every YAIB deep learning model receives, so its results are not comparable to those reported for EHRMamba:

  • Input. EHRMamba reads a sequence of tokenized medical codes and events with concept, type, time, age and visit embeddings. YAIB gives each stay a dense hourly grid of features (with missingness indicators), so a linear layer embeds each hour, as in YAIB's Transformer. No positional or time embedding is added: the recurrence is ordered by construction, and the hourly grid makes the time between steps constant.
  • Training. EHRMamba is pretrained on all of MIMIC-IV and then fine-tuned with multitask prompted fine-tuning. Here the model is trained from scratch for each task, like the other YAIB deep learning models.
  • Output. EHRMamba classifies a whole sequence. YAIB predicts at every hour, so the model returns one prediction per time step. Mamba is causal (the convolution is left-padded and the scan runs forward in time), so the prediction at hour t only depends on hours up to t and per-stay tasks use the prediction at the last observed hour. YAIB pads sequences at the end, so padding never affects the predictions for real hours.
  • Size. EHRMamba uses 32 layers with a width of 768 on sequences of up to 2048 tokens. YAIB's sequences are at most a few hundred hours, so the default configuration is much smaller (see configs/prediction_models/Mamba.gin).

The layer itself follows the reference implementation (https://github.com/state-spaces/mamba, Mamba and MixerModel): input projection with a gate, causal depthwise convolution, input-dependent step size and B/C projections, S4D-real initialization of A, the inverse-softplus initialization of the step-size bias, pre-norm residual blocks with RMSNorm, and output projections rescaled by 1/sqrt(depth). The scan runs in plain PyTorch, so the model trains on CPU, MPS and CUDA without extra packages. If mamba-ssm is installed (pip install mamba-ssm, which needs an NVIDIA GPU) and the model runs on CUDA, its fused selective-scan kernel is used instead; the results are the same up to floating-point differences.

Classes:

Name Description
MambaMixer

A Mamba layer: gated selective state-space model with a causal depthwise convolution.

MambaBlock

Pre-norm residual block around a Mamba layer.

Mamba

Mamba selective state-space model with one prediction per time step.

Functions:

Name Description
selective_scan

Run the selective state-space recurrence in plain PyTorch.

use_fused_scan

Whether to use the fused CUDA kernel from mamba-ssm for this input.

MambaMixer

MambaMixer(d_model, d_state=16, d_conv=4, expand=2, dt_rank='auto', dt_min=0.001, dt_max=0.1, dt_init_floor=0.0001)

Bases: Module

A Mamba layer: gated selective state-space model with a causal depthwise convolution.

Source code in icu_benchmarks/models/dl_models/mamba.py
def __init__(
    self,
    d_model,
    d_state=16,
    d_conv=4,
    expand=2,
    dt_rank="auto",
    dt_min=0.001,
    dt_max=0.1,
    dt_init_floor=1e-4,
):
    super().__init__()
    self.d_inner = int(expand * d_model)
    self.d_state = d_state
    self.dt_rank = math.ceil(d_model / 16) if dt_rank == "auto" else dt_rank

    self.in_proj = nn.Linear(d_model, 2 * self.d_inner, bias=False)
    self.conv1d = nn.Conv1d(self.d_inner, self.d_inner, d_conv, groups=self.d_inner, padding=d_conv - 1)
    self.x_proj = nn.Linear(self.d_inner, self.dt_rank + 2 * d_state, bias=False)
    self.dt_proj = nn.Linear(self.dt_rank, self.d_inner, bias=True)
    self.out_proj = nn.Linear(self.d_inner, d_model, bias=False)

    # Step sizes start log-uniform in [dt_min, dt_max]; the bias is their inverse softplus
    dt_init_std = self.dt_rank**-0.5
    nn.init.uniform_(self.dt_proj.weight, -dt_init_std, dt_init_std)
    dt = torch.exp(torch.rand(self.d_inner) * (math.log(dt_max) - math.log(dt_min)) + math.log(dt_min))
    dt = dt.clamp(min=dt_init_floor)
    with torch.no_grad():
        self.dt_proj.bias.copy_(dt + torch.log(-torch.expm1(-dt)))

    # S4D-real initialization: A = -(1, 2, ..., d_state) for every channel, stored as log(-A) to keep A negative
    A = torch.arange(1, d_state + 1, dtype=torch.float32).repeat(self.d_inner, 1)
    self.A_log = nn.Parameter(torch.log(A))
    self.D = nn.Parameter(torch.ones(self.d_inner))

MambaBlock

MambaBlock(d_model, dropout=0.0, **mixer_kwargs)

Bases: Module

Pre-norm residual block around a Mamba layer.

Source code in icu_benchmarks/models/dl_models/mamba.py
def __init__(self, d_model, dropout=0.0, **mixer_kwargs):
    super().__init__()
    self.norm = nn.RMSNorm(d_model, eps=1e-5)
    self.mixer = MambaMixer(d_model, **mixer_kwargs)
    self.dropout = nn.Dropout(dropout)

Mamba

Mamba(input_size, hidden, depth, num_classes, *args, d_state=16, d_conv=4, expand=2, dropout=0.0, **kwargs)

Bases: DLPredictionWrapper

Mamba selective state-space model with one prediction per time step.

Parameters:

Name Type Description Default
input_size

Shape of an input batch (batch, time, features).

required
hidden

Width of the model (d_model).

required
depth

Number of Mamba blocks.

required
num_classes

Number of outputs per time step.

required
d_state

Size of the state per channel.

16
d_conv

Kernel size of the causal convolution.

4
expand

Expansion factor of the inner channels (d_inner = expand * hidden).

2
dropout

Dropout on the embedded input and on each block's output.

0.0
Source code in icu_benchmarks/models/dl_models/mamba.py
def __init__(
    self,
    input_size,
    hidden,
    depth,
    num_classes,
    *args,
    d_state=16,
    d_conv=4,
    expand=2,
    dropout=0.0,
    **kwargs,
):
    super().__init__(
        *args,
        input_size=input_size,
        hidden=hidden,
        depth=depth,
        num_classes=num_classes,
        d_state=d_state,
        d_conv=d_conv,
        expand=expand,
        dropout=dropout,
        **kwargs,
    )
    self.input_embedding = nn.Linear(input_size[2], hidden)
    self.embedding_dropout = nn.Dropout(dropout)
    self.blocks = nn.Sequential(
        *[MambaBlock(hidden, dropout=dropout, d_state=d_state, d_conv=d_conv, expand=expand) for _ in range(depth)]
    )
    self.norm = nn.RMSNorm(hidden, eps=1e-5)
    self.logit = nn.Linear(hidden, num_classes)
    # As in the reference MixerModel: keep the residual stream's variance from growing with depth
    for block in self.blocks:
        with torch.no_grad():
            block.mixer.out_proj.weight /= math.sqrt(depth)

selective_scan

selective_scan(u, delta, A, B, C, D)

Run the selective state-space recurrence in plain PyTorch.

h_t = exp(delta_t * A) * h_{t-1} + delta_t * B_t * u_t and y_t = C_t . h_t + D * u_t, for every channel.

Parameters:

Name Type Description Default
u

Input of shape (batch, time, channels).

required
delta

Positive step sizes of shape (batch, time, channels).

required
A

State matrix (diagonal, negative) of shape (channels, state).

required
B

Input matrices of shape (batch, time, state).

required
C

Output matrices of shape (batch, time, state).

required
D

Skip connection of shape (channels,).

required

Returns:

Type Description

Output of shape (batch, time, channels).

Source code in icu_benchmarks/models/dl_models/mamba.py
def selective_scan(u, delta, A, B, C, D):
    """Run the selective state-space recurrence in plain PyTorch.

    h_t = exp(delta_t * A) * h_{t-1} + delta_t * B_t * u_t and y_t = C_t . h_t + D * u_t, for every channel.

    Args:
        u: Input of shape (batch, time, channels).
        delta: Positive step sizes of shape (batch, time, channels).
        A: State matrix (diagonal, negative) of shape (channels, state).
        B: Input matrices of shape (batch, time, state).
        C: Output matrices of shape (batch, time, state).
        D: Skip connection of shape (channels,).

    Returns:
        Output of shape (batch, time, channels).
    """
    batch, _, channels = u.shape
    # (batch, channels, state) per time step. Unbinding once instead of indexing every step keeps the backward pass
    # linear in the sequence length: each index would allocate a gradient the size of the whole sequence.
    decay = torch.exp(delta.unsqueeze(-1) * A).unbind(1)
    update = ((delta * u).unsqueeze(-1) * B.unsqueeze(2)).unbind(1)
    state = u.new_zeros(batch, channels, A.shape[1])
    states = []
    for step_decay, step_update in zip(decay, update):
        state = torch.addcmul(step_update, step_decay, state)
        states.append(state)
    return torch.einsum("btcs,bts->btc", torch.stack(states, dim=1), C) + u * D

use_fused_scan

use_fused_scan(u)

Whether to use the fused CUDA kernel from mamba-ssm for this input.

Source code in icu_benchmarks/models/dl_models/mamba.py
def use_fused_scan(u):
    """Whether to use the fused CUDA kernel from mamba-ssm for this input."""
    return selective_scan_fn is not None and u.is_cuda

icu_benchmarks.models.dl_models.layers

Classes:

Name Description
PositionalEncoding
SelfAttention

Multi Head Attention block from Attention is All You Need (https://arxiv.org/abs/1706.03762). Input has shape

Chomp1d

From TCN original paper https://github.com/locuslab/TCN

Functions:

Name Description
parallel_recomb

Return mask of attention matrix (ts_q, ts_kv)

PositionalEncoding

PositionalEncoding(emb, max_len=3000)

Bases: Module

Positional Encoding, mostly from: https://pytorch.org/tutorials/beginner/transformer_tutorial.html

Source code in icu_benchmarks/models/dl_models/layers.py
def __init__(self, emb, max_len=3000):
    super().__init__()
    pe = torch.zeros(max_len, emb)
    position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, emb, 2).float() * (-math.log(10000.0) / emb))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    pe = pe.unsqueeze(0)
    self.register_buffer("pe", pe)

SelfAttention

SelfAttention(emb, hidden, heads=8, mask=True, att_type='all', local_context=None, mask_aggregation='union', dropout_att=0.0)

Bases: Module

Multi Head Attention block from Attention is All You Need (https://arxiv.org/abs/1706.03762). Input has shape (batch_size, n_timestamps, emb).

Initialize the Multi Head Block. Args: emb: Dimension of the input vector. hidden: Hidden dimension of query, key, value matrices. heads: Number of heads. mask: Whether to mask the attention matrix.

Methods:

Name Description
forward

Args:

Source code in icu_benchmarks/models/dl_models/layers.py
def __init__(
    self,
    emb,
    hidden,
    heads=8,
    mask=True,
    att_type="all",
    local_context=None,
    mask_aggregation="union",
    dropout_att=0.0,
):
    """Initialize the Multi Head Block.
    Args:
        emb: Dimension of the input vector.
        hidden: Hidden dimension of query, key, value matrices.
        heads: Number of heads.
        mask: Whether to mask the attention matrix."""
    super().__init__()

    self.emb = emb
    self.heads = heads
    self.hidden = hidden
    self.mask = mask
    self.drop_att = nn.Dropout(dropout_att)

    # Sparse transformer specific params
    self.att_type = att_type
    self.local_context = local_context
    self.mask_aggregation = mask_aggregation

    # Query, keys and value matrices
    self.w_keys = nn.Linear(emb, hidden * heads, bias=False)
    self.w_queries = nn.Linear(emb, hidden * heads, bias=False)
    self.w_values = nn.Linear(emb, hidden * heads, bias=False)

    # Output linear function
    self.unifyheads = nn.Linear(heads * hidden, emb)

forward

forward(x)

Parameters:

Name Type Description Default
x

Input data tensor with shape (batch_size, n_timestamps, emb)

required

Returns: Self attention tensor with shape (batch_size, n_timestamps, emb)

Source code in icu_benchmarks/models/dl_models/layers.py
def forward(self, x):
    """
    Args:
        x: Input data tensor with shape (batch_size, n_timestamps, emb)
    Returns:
        Self attention tensor with shape (batch_size, n_timestamps, emb)
    """
    # bs - batch_size, n - vectors number, emb - embedding dimensionality
    bs, n, emb = x.size()
    h = self.heads
    hidden = self.hidden

    keys = self.w_keys(x).view(bs, n, h, hidden)
    queries = self.w_queries(x).view(bs, n, h, hidden)
    values = self.w_values(x).view(bs, n, h, hidden)

    # fold heads into the batch dimension
    keys = keys.transpose(1, 2).contiguous().view(bs * h, n, hidden)
    queries = queries.transpose(1, 2).contiguous().view(bs * h, n, hidden)
    values = values.transpose(1, 2).contiguous().view(bs * h, n, hidden)

    # dive on the square oot of dimensionality
    queries = queries / (hidden ** (1 / 2))
    keys = keys / (hidden ** (1 / 2))

    # dot product of queries and keys, and scale
    dot = torch.bmm(queries, keys.transpose(1, 2))

    if self.mask:  # We deal with different masking and recombination types here
        if isinstance(self.att_type, list):  # Local and sparse attention
            if self.mask_aggregation == "union":
                mask_tensor = 0
                for att_type in self.att_type:
                    mask_tensor += parallel_recomb(
                        torch.arange(1, n + 1, dtype=torch.float, device=dot.device).reshape(1, -1),
                        torch.arange(1, n + 1, dtype=torch.float, device=dot.device).reshape(1, -1),
                        att_type,
                        self.local_context,
                    )[0]
                mask_tensor = torch.clamp(mask_tensor, 0, 1)
                dot = torch.where(
                    mask_tensor.bool(),
                    dot,
                    torch.tensor(float("-inf")).to(dot.device),
                ).view(bs * h, n, n)

            elif self.mask_aggregation == "split":
                dot_list = list(torch.split(dot, dot.shape[0] // len(self.att_type), dim=0))
                for i, att_type in enumerate(self.att_type):
                    mask_tensor = parallel_recomb(
                        torch.arange(1, n + 1, dtype=torch.float, device=dot.device).reshape(1, -1),
                        torch.arange(1, n + 1, dtype=torch.float, device=dot.device).reshape(1, -1),
                        att_type,
                        self.local_context,
                    )[0]

                    dot_list[i] = torch.where(
                        mask_tensor.bool(),
                        dot_list[i],
                        torch.tensor(float("-inf")).to(dot.device),
                    ).view(*dot_list[i].shape)
                dot = torch.cat(dot_list, dim=0)
        else:  # Full causal masking
            mask_tensor = parallel_recomb(
                torch.arange(1, n + 1, dtype=torch.float, device=dot.device).reshape(1, -1),
                torch.arange(1, n + 1, dtype=torch.float, device=dot.device).reshape(1, -1),
                self.att_type,
                self.local_context,
            )[0]
            dot = torch.where(mask_tensor.bool(), dot, torch.tensor(float("-inf")).to(dot.device)).view(bs * h, n, n)

    # dot now has row-wise self-attention probabilities
    dot = F.softmax(dot, dim=2)

    # apply the self attention to the values
    out = torch.bmm(dot, values).view(bs, h, n, hidden)

    # apply the dropout
    out = self.drop_att(out)

    # swap h, t back, unify heads
    out = out.transpose(1, 2).contiguous().view(bs, n, h * hidden)
    return self.unifyheads(out)

Chomp1d

Chomp1d(chomp_size)

Bases: Module

From TCN original paper https://github.com/locuslab/TCN

Source code in icu_benchmarks/models/dl_models/layers.py
def __init__(self, chomp_size):
    super(Chomp1d, self).__init__()
    self.chomp_size = chomp_size

parallel_recomb

parallel_recomb(q_t, kv_t, att_type='all', local_context=3, bin_size=None)

Return mask of attention matrix (ts_q, ts_kv)

Source code in icu_benchmarks/models/dl_models/layers.py
@gin.configurable("masking")
def parallel_recomb(q_t, kv_t, att_type="all", local_context=3, bin_size=None):
    """Return mask of attention matrix (ts_q, ts_kv)"""
    with torch.no_grad():
        q_t[q_t == -1.0] = float("inf")  # We want padded to attend to everyone to avoid any nan.
        kv_t[kv_t == -1.0] = float("inf")  # We want no one to attend the padded values

        if bin_size is not None:  # General case where we use unaligned timesteps.
            q_t = q_t / bin_size
            starts_q = q_t[:, 0:1].clone()  # Needed because of Memory allocation issue
            q_t -= starts_q
            kv_t = kv_t / bin_size
            starts_kv = kv_t[:, 0:1].clone()  # Needed because of Memory allocation issue
            kv_t -= starts_kv

        bs, ts_q = q_t.size()
        _, ts_kv = kv_t.size()
        q_t_rep = q_t.view(bs, ts_q, 1).repeat(1, 1, ts_kv)
        kv_t_rep = kv_t.view(bs, 1, ts_kv).repeat(1, ts_q, 1)
        diff_mask = (q_t_rep - kv_t_rep).to(q_t_rep.device)
        if att_type == "all":
            return (diff_mask >= 0).float()
        if att_type == "local":
            return ((diff_mask >= 0) * (diff_mask <= local_context) + (diff_mask == float("inf"))).float()
        if att_type == "strided":
            return ((diff_mask >= 0) * (torch.floor(diff_mask) % local_context == 0) + (diff_mask == float("inf"))).float()