Deep learning models¶
PyTorch Lightning prediction models. Each is a gin configurable named in its configs/prediction_models/*.gin file.
icu_benchmarks.models.dl_models.rnn
¶
Classes:
| Name | Description |
|---|---|
RNNet |
Torch standard RNN model |
LSTMNet |
Torch standard LSTM model. |
GRUNet |
Torch standard GRU model. |
RNNet
¶
Bases: DLPredictionWrapper
Torch standard RNN model
Source code in icu_benchmarks/models/dl_models/rnn.py
LSTMNet
¶
Bases: DLPredictionWrapper
Torch standard LSTM model.
Source code in icu_benchmarks/models/dl_models/rnn.py
GRUNet
¶
Bases: DLPredictionWrapper
Torch standard GRU model.
Source code in icu_benchmarks/models/dl_models/rnn.py
icu_benchmarks.models.dl_models.grud
¶
GRU-D: a GRU that decays inputs and hidden state with the time since each variable was last observed.
GRU-D (Che et al., Recurrent Neural Networks for Multivariate Time Series with Missing Values, Scientific Reports 8, 6085, 2018, https://doi.org/10.1038/s41598-018-24271-9) feeds a GRU the observation mask m_t and decays both the imputed inputs and the hidden state with the time delta_t since each variable was last observed (equations 2 and 10 to 16 of the paper):
gamma_x = exp(-max(0, W_gamma_x delta_t + b_gamma_x)) (W_gamma_x diagonal)
gamma_h = exp(-max(0, W_gamma_h delta_t + b_gamma_h))
x_hat_t = m_t x_t + (1 - m_t) (gamma_x x_last + (1 - gamma_x) x_mean)
h_hat = gamma_h h_{t-1}
z_t, r_t = sigmoid(W x_hat_t + U h_hat + V m_t + b) (one set of weights per gate)
h_t = (1 - z_t) h_hat + z_t tanh(W x_hat_t + U (r_t h_hat) + V m_t + b)
Ported from the YAIB fork by Katja Jagd and Rachael DeVries (https://github.com/Katja-Jagd/YAIB, branch katja,
icu_benchmarks/models/dl_models/grud.py, MIT License), which was used for BAT pretraining on YAIB (Jagd et al.,
https://arxiv.org/abs/2509.19885) and as a baseline for the Bi-Axial Transformer (DeVries et al.,
https://arxiv.org/abs/2508.12418). The fork's cell is a PyTorch translation of the TensorFlow GRU-D in SeFT (Horn et al.,
https://github.com/BorgwardtLab/Set_Functions_for_Time_Series, seft/models/gru_d.py, BSD 3-Clause License), which
adapts the Keras implementation by the first author of GRU-D (https://github.com/PeterChe1990/GRU-D,
nn_utils/grud_layers.py, MIT License). Both license notices are reproduced at the end of this docstring.
How the port fits YAIB:
- Inputs. The model takes the same hourly feature matrix as YAIB's GRU. The observation mask m_t comes from the
MissingIndicator_<name>columns of the preprocessing recipe (seeObservationMaskWrapper); columns without an indicator, including the indicators themselves, count as always observed. YAIB's data is on a regular hourly grid, so delta_t counts hours since the last measurement. The features are standardised, so the empirical mean x_mean is 0, which is also the value the recipe fills in before the first measurement. Static features are inputs at every hour, as for the other YAIB models, instead of initialising the hidden state as in the fork and SeFT. - Output. The model predicts at every hour from the hidden state h_t, so it is causal by construction. Per-stay
tasks use the prediction at the last observed hour, which is the fork's
pooling="hidden". - Fixes to the fork. The decay parameters were initialised to exactly zero, where relu has no gradient, so the
decays never trained; their biases now start slightly positive (decay rates close to 1, as before). The gate biases
were never initialised and were not used when the mask was fed to the gates (the Keras reference always adds them).
Recurrent and mask dropout were never applied because of an inverted condition. The new state interpolates with the
decayed state h_hat, as in equation 16 of the paper (the Keras reference used the undecayed h_{t-1}). The cell is a
plain module instead of TorchScript, so the model can be pickled with
save_model. - Left out. The fork's options for ablations and other tasks: zero, forward or raw imputation instead of the
input decay, masking decay, indicator-only or value-only inputs, mean or max pooling over time, the static encoder,
self-supervised pretraining (
SSL_GRUD) and GRU-D as an imputation model.
License notices of the upstream implementations:
SeFT (https://github.com/BorgwardtLab/Set_Functions_for_Time_Series), BSD 3-Clause License:
Copyright 2020 Max Horn
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AD CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
GRU-D Keras implementation (https://github.com/PeterChe1990/GRU-D), MIT License:
Copyright (c) 2018 Zhengping Che
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of
the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO
THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Classes:
| Name | Description |
|---|---|
GRUDLayer |
GRU-D recurrence over a whole sequence. |
GRUD |
GRU-D (Che et al., 2018) with one prediction per hour; see the module docstring. |
Functions:
| Name | Description |
|---|---|
decay_rate |
Decay rate exp(-max(0, x)) in (0, 1], monotonically decreasing in the time since the last observation. |
variational_dropout |
One dropout mask per sequence, shared by all time steps (Gal and Ghahramani, 2016), as in the Keras GRU-D. |
GRUDLayer
¶
Bases: Module
GRU-D recurrence over a whole sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_dim
|
int
|
Number of input columns. |
required |
hidden_dim
|
int
|
Size of the hidden state. |
required |
dropout
|
float
|
Dropout on the inputs and the mask, with one mask per sequence. |
0.0
|
recurrent_dropout
|
float
|
Dropout on the hidden state in the recurrent connections, with one mask per sequence. |
0.0
|
Methods:
| Name | Description |
|---|---|
reset_parameters |
Initialise as the Keras reference: Glorot-uniform kernels, orthogonal recurrent kernels, zero gate biases. |
forward |
Run GRU-D over the sequence. |
Source code in icu_benchmarks/models/dl_models/grud.py
reset_parameters
¶
Initialise as the Keras reference: Glorot-uniform kernels, orthogonal recurrent kernels, zero gate biases.
The decays start without decay, as in the reference, but with a small positive bias instead of zero: relu has no gradient at zero, so all-zero decay parameters would never train.
Source code in icu_benchmarks/models/dl_models/grud.py
forward
¶
Run GRU-D over the sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Tensor
|
Inputs of shape (batch, time, columns). Values that were not observed are ignored, except as a fallback for the first hours before a column was measured, where the empirical mean 0 is used. |
required |
observed
|
Tensor
|
Observation mask of the same shape, 1 where x was measured and 0 where it was imputed. |
required |
Returns:
| Type | Description |
|---|---|
Tensor
|
Hidden states of shape (batch, time, hidden_dim). |
Source code in icu_benchmarks/models/dl_models/grud.py
GRUD
¶
Bases: ObservationMaskWrapper
GRU-D (Che et al., 2018) with one prediction per hour; see the module docstring.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_size
|
Shape (batch, time, columns) of the input. |
required | |
hidden_dim
|
Size of the hidden state. |
required | |
num_classes
|
Number of classes, or 1 for regression. |
required | |
dropout
|
Dropout on the inputs and the observation mask, with one mask per sequence. |
0.0
|
|
recurrent_dropout
|
Dropout on the hidden state in the recurrent connections, with one mask per sequence. |
0.0
|
Source code in icu_benchmarks/models/dl_models/grud.py
decay_rate
¶
Decay rate exp(-max(0, x)) in (0, 1], monotonically decreasing in the time since the last observation.
variational_dropout
¶
One dropout mask per sequence, shared by all time steps (Gal and Ghahramani, 2016), as in the Keras GRU-D.
Source code in icu_benchmarks/models/dl_models/grud.py
icu_benchmarks.models.dl_models.tcn
¶
Classes:
| Name | Description |
|---|---|
TemporalConvNet |
Temporal Convolutional Network. Adapted from TCN original paper https://github.com/locuslab/TCN |
TemporalConvNet
¶
TemporalConvNet(input_size, num_channels, num_classes, *args, max_seq_length=0, kernel_size=2, dropout=0.0, **kwargs)
Bases: DLPredictionWrapper
Temporal Convolutional Network. Adapted from TCN original paper https://github.com/locuslab/TCN
Source code in icu_benchmarks/models/dl_models/tcn.py
icu_benchmarks.models.dl_models.transformer
¶
Classes:
| Name | Description |
|---|---|
BaseTransformer |
|
Transformer |
Transformer model. |
LocalTransformer |
Transformer model with local context. |
BaseTransformer
¶
BaseTransformer(block_class, input_size, hidden, heads, ff_hidden_mult, depth, num_classes, dropout=0.0, pos_encoding=True, dropout_att=0.0, local_context=None, *args, **kwargs)
Bases: DLPredictionWrapper
Source code in icu_benchmarks/models/dl_models/transformer.py
Transformer
¶
Bases: BaseTransformer
Transformer model.
Source code in icu_benchmarks/models/dl_models/transformer.py
LocalTransformer
¶
Bases: BaseTransformer
Transformer model with local context.
Source code in icu_benchmarks/models/dl_models/transformer.py
icu_benchmarks.models.dl_models.bat
¶
Bi-Axial Transformer (BAT): attention along both the variable axis and the time axis of the hourly grid.
BAT (DeVries et al., Bi-Axial Transformers: Addressing the Increasing Complexity of EHR Classification, 2025, https://arxiv.org/abs/2508.12418) turns every (variable, hour) cell of a stay into a token made of the value, whether it was measured, a learned embedding of the variable and an encoding of the time. Two streams of these tokens attend along both axes in opposite orders: one attends across the variables within each hour and then across the hours of each variable, the other across the hours first. Both streams use the same attention layers for each axis, are pooled over the variables and merged by a nonlinear layer. Jagd et al. (https://arxiv.org/abs/2509.19885) pretrained BAT on YAIB.
Ported from the YAIB fork by Katja Jagd and Rachael DeVries (https://github.com/Katja-Jagd/YAIB, branch katja,
icu_benchmarks/models/dl_models/bat.py, MIT License). The fork's BAT encoder comes from the BAT authors' reference
code, and its causal ("autoregressive") encoder, which this module ports, was added by Rachael DeVries. The time encoding
is SeFT's positional encoding (Horn et al., https://github.com/BorgwardtLab/Set_Functions_for_Time_Series,
seft/models/transformer.py, BSD 3-Clause License); its license notice is reproduced at the end of this docstring.
How the port fits YAIB:
- Inputs. The variables ("sensors") are the input columns that are not missingness indicators, static features
included. Each token gets the value and the observation flag read from the
MissingIndicator_<name>column (seeObservationMaskWrapper); variables without an indicator count as measured. YAIB's data is on a regular hourly grid, so the time of a token is its hour in the stay. Static features are variables like the others instead of a separate embedding that is concatenated after pooling, as in the fork. - Output. Attention across hours is causal and every other step works within an hour, so the prediction for hour t only depends on hours up to t. The model predicts at every hour, and per-stay tasks use the prediction at the last observed hour, which has attended to the whole stay. YAIB pads stays at the end, so padding never affects the predictions for real hours and no padding mask is needed.
- Not ported. The bidirectional encoder of the BAT paper, which max-pools over all variables and hours, is not
included: it can only serve per-stay tasks, and YAIB does not pass a padding mask to
forward, so it would have to guess the padding from all-zero rows. The fork used it for per-stay tasks and the causal encoder for hourly tasks. Also left out: the pooling and input ablation options, attention-map outputs, the separate prediction heads (thelogitlayer replaces them), the forecasting head and self-supervised pretraining (SSL_BAT). - Attention layers. YAIB's
TransformerBlock(post-norm, ReLU feed-forward) replaces the pre-norm blocks of the x-transformers package, which avoids a new dependency. Hyperparameters follow YAIB's Transformer:depthis the fork'slayersanddropout_attitsattn_dropout. - Fixes to the fork. The padding mask of the time attention (
use_mask) was repeated in the wrong order, so each row of a batch got the mask of another stay; it is no longer needed. An oddvalue_embed_sizebroke the time encoding; it is rounded up to an even size. The placeholderlogitlayer, which only existed to select the metrics, is now the output layer.
License notice of SeFT (https://github.com/BorgwardtLab/Set_Functions_for_Time_Series), BSD 3-Clause License:
Copyright 2020 Max Horn
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AD CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Classes:
| Name | Description |
|---|---|
BAT |
Causal Bi-Axial Transformer with one prediction per hour; see the module docstring. |
Functions:
| Name | Description |
|---|---|
time_encoding |
SeFT's sinusoidal encoding of the times 0, ..., steps - 1, with timescales from 1 to max_time. |
BAT
¶
BAT(input_size, num_classes, value_embed_size=8, depth=1, heads=1, ff_hidden_mult=4, dropout=0.0, dropout_att=0.0, max_time=500, *args, **kwargs)
Bases: ObservationMaskWrapper
Causal Bi-Axial Transformer with one prediction per hour; see the module docstring.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_size
|
Shape (batch, time, columns) of the input. |
required | |
num_classes
|
Number of classes, or 1 for regression. |
required | |
value_embed_size
|
Size of the embedding of a value and its observation flag (rounded up to an even number). The token size adds an embedding of the variable, of size 1.6 * sqrt(input columns) (the fork used the number of variables, which is only known once the column names are set). |
8
|
|
depth
|
Number of attention blocks per axis. |
1
|
|
heads
|
Number of attention heads. |
1
|
|
ff_hidden_mult
|
Width of the feed-forward layers, as a multiple of the token size. |
4
|
|
dropout
|
Dropout in the attention blocks. |
0.0
|
|
dropout_att
|
Dropout on the attention outputs. |
0.0
|
|
max_time
|
Largest timescale of the time encoding, in hours. |
500
|
Source code in icu_benchmarks/models/dl_models/bat.py
time_encoding
¶
SeFT's sinusoidal encoding of the times 0, ..., steps - 1, with timescales from 1 to max_time.
Returns:
| Type | Description |
|---|---|
Tensor
|
Tensor of shape (steps, dim): sines in the first half and cosines in the second half of the last axis. |
Source code in icu_benchmarks/models/dl_models/bat.py
icu_benchmarks.models.dl_models.mamba
¶
Mamba selective state-space model for YAIB's hourly time series.
Mamba (Gu and Dao, 2023, https://arxiv.org/abs/2312.00752) replaces attention with a selective state-space layer whose state update depends on the input, so it scales linearly with sequence length. EHRMamba (Fallahpour et al., 2024, https://arxiv.org/abs/2405.14567) showed that this backbone works well for EHR prediction. This module contains the Mamba backbone only, adapted to the inputs every YAIB deep learning model receives, so its results are not comparable to those reported for EHRMamba:
- Input. EHRMamba reads a sequence of tokenized medical codes and events with concept, type, time, age and visit embeddings. YAIB gives each stay a dense hourly grid of features (with missingness indicators), so a linear layer embeds each hour, as in YAIB's Transformer. No positional or time embedding is added: the recurrence is ordered by construction, and the hourly grid makes the time between steps constant.
- Training. EHRMamba is pretrained on all of MIMIC-IV and then fine-tuned with multitask prompted fine-tuning. Here the model is trained from scratch for each task, like the other YAIB deep learning models.
- Output. EHRMamba classifies a whole sequence. YAIB predicts at every hour, so the model returns one prediction per time step. Mamba is causal (the convolution is left-padded and the scan runs forward in time), so the prediction at hour t only depends on hours up to t and per-stay tasks use the prediction at the last observed hour. YAIB pads sequences at the end, so padding never affects the predictions for real hours.
- Size. EHRMamba uses 32 layers with a width of 768 on sequences of up to 2048 tokens. YAIB's sequences are at most a few hundred hours, so the default configuration is much smaller (see configs/prediction_models/Mamba.gin).
The layer itself follows the reference implementation (https://github.com/state-spaces/mamba, Mamba and
MixerModel): input projection with a gate, causal depthwise convolution, input-dependent step size and B/C
projections, S4D-real initialization of A, the inverse-softplus initialization of the step-size bias, pre-norm residual
blocks with RMSNorm, and output projections rescaled by 1/sqrt(depth). The scan runs in plain PyTorch, so the model
trains on CPU, MPS and CUDA without extra packages. If mamba-ssm is installed (pip install mamba-ssm, which
needs an NVIDIA GPU) and the model runs on CUDA, its fused selective-scan kernel is used instead; the results are the
same up to floating-point differences.
Classes:
| Name | Description |
|---|---|
MambaMixer |
A Mamba layer: gated selective state-space model with a causal depthwise convolution. |
MambaBlock |
Pre-norm residual block around a Mamba layer. |
Mamba |
Mamba selective state-space model with one prediction per time step. |
Functions:
| Name | Description |
|---|---|
selective_scan |
Run the selective state-space recurrence in plain PyTorch. |
use_fused_scan |
Whether to use the fused CUDA kernel from mamba-ssm for this input. |
MambaMixer
¶
MambaMixer(d_model, d_state=16, d_conv=4, expand=2, dt_rank='auto', dt_min=0.001, dt_max=0.1, dt_init_floor=0.0001)
Bases: Module
A Mamba layer: gated selective state-space model with a causal depthwise convolution.
Source code in icu_benchmarks/models/dl_models/mamba.py
MambaBlock
¶
Bases: Module
Pre-norm residual block around a Mamba layer.
Source code in icu_benchmarks/models/dl_models/mamba.py
Mamba
¶
Mamba(input_size, hidden, depth, num_classes, *args, d_state=16, d_conv=4, expand=2, dropout=0.0, **kwargs)
Bases: DLPredictionWrapper
Mamba selective state-space model with one prediction per time step.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
input_size
|
Shape of an input batch (batch, time, features). |
required | |
hidden
|
Width of the model (d_model). |
required | |
depth
|
Number of Mamba blocks. |
required | |
num_classes
|
Number of outputs per time step. |
required | |
d_state
|
Size of the state per channel. |
16
|
|
d_conv
|
Kernel size of the causal convolution. |
4
|
|
expand
|
Expansion factor of the inner channels (d_inner = expand * hidden). |
2
|
|
dropout
|
Dropout on the embedded input and on each block's output. |
0.0
|
Source code in icu_benchmarks/models/dl_models/mamba.py
selective_scan
¶
Run the selective state-space recurrence in plain PyTorch.
h_t = exp(delta_t * A) * h_{t-1} + delta_t * B_t * u_t and y_t = C_t . h_t + D * u_t, for every channel.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
u
|
Input of shape (batch, time, channels). |
required | |
delta
|
Positive step sizes of shape (batch, time, channels). |
required | |
A
|
State matrix (diagonal, negative) of shape (channels, state). |
required | |
B
|
Input matrices of shape (batch, time, state). |
required | |
C
|
Output matrices of shape (batch, time, state). |
required | |
D
|
Skip connection of shape (channels,). |
required |
Returns:
| Type | Description |
|---|---|
|
Output of shape (batch, time, channels). |
Source code in icu_benchmarks/models/dl_models/mamba.py
use_fused_scan
¶
icu_benchmarks.models.dl_models.layers
¶
Classes:
| Name | Description |
|---|---|
PositionalEncoding |
Positional Encoding, mostly from: https://pytorch.org/tutorials/beginner/transformer_tutorial.html |
SelfAttention |
Multi Head Attention block from Attention is All You Need (https://arxiv.org/abs/1706.03762). Input has shape |
Chomp1d |
From TCN original paper https://github.com/locuslab/TCN |
Functions:
| Name | Description |
|---|---|
parallel_recomb |
Return mask of attention matrix (ts_q, ts_kv) |
PositionalEncoding
¶
Bases: Module
Positional Encoding, mostly from: https://pytorch.org/tutorials/beginner/transformer_tutorial.html
Source code in icu_benchmarks/models/dl_models/layers.py
SelfAttention
¶
SelfAttention(emb, hidden, heads=8, mask=True, att_type='all', local_context=None, mask_aggregation='union', dropout_att=0.0)
Bases: Module
Multi Head Attention block from Attention is All You Need (https://arxiv.org/abs/1706.03762). Input has shape (batch_size, n_timestamps, emb).
Initialize the Multi Head Block. Args: emb: Dimension of the input vector. hidden: Hidden dimension of query, key, value matrices. heads: Number of heads. mask: Whether to mask the attention matrix.
Methods:
| Name | Description |
|---|---|
forward |
Args: |
Source code in icu_benchmarks/models/dl_models/layers.py
forward
¶
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
x
|
Input data tensor with shape (batch_size, n_timestamps, emb) |
required |
Returns: Self attention tensor with shape (batch_size, n_timestamps, emb)
Source code in icu_benchmarks/models/dl_models/layers.py
97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 | |
Chomp1d
¶
Bases: Module
From TCN original paper https://github.com/locuslab/TCN
Source code in icu_benchmarks/models/dl_models/layers.py
parallel_recomb
¶
Return mask of attention matrix (ts_q, ts_kv)