Skip to content

Imputation models

Imputation models and the amputation methods used to train and evaluate them. See Imputation.

icu_benchmarks.imputation.amputations

This file implements amputation mechanisms (MCAR, MAR (logisitc) and MNAR (logistic)) for missing data generation. It was inspired from: https://rmisstastic.netlify.app/how-to/python/generate_html/how%20to%20generate%20missing%20values Original code: https://github.com/BorisMuzellec/MissingDataOT/blob/master/utils.py

Functions:

Name Description
MCAR_mask

Missing completely at random mechanism.

BO_mask

Black out missing mechanism. Removes values across dimensions.

MAR_logistic_mask

Missing at random mechanism with a logistic masking model. First, a subset of variables with no missing values is

MNAR_logistic_mask

Missing not at random mechanism with a logistic masking model. It implements two mechanisms:

ampute_data

Generate missing values for specifics missing-data mechanism and proportion of missing values.

MCAR_mask

MCAR_mask(X, p)

Missing completely at random mechanism.

Parameters

X : torch.FloatTensor, shape (n, d) Data for which missing values will be simulated. p : float Proportion of missing values to generate for variables which will have missing values.

Returns

mask : torch.BoolTensor Mask of generated missing values (True if the value is missing).

Source code in icu_benchmarks/imputation/amputations.py
def MCAR_mask(X, p):
    """
    Missing completely at random mechanism.

    Parameters
    ----------
    X : torch.FloatTensor, shape (n, d)
        Data for which missing values will be simulated.
    p : float
        Proportion of missing values to generate for variables which will have missing values.

    Returns
    -------
    mask : torch.BoolTensor
        Mask of generated missing values (True if the value is missing).
    """

    n, d = X.shape
    mask = np.zeros((n, d))

    ber = torch.rand(n, d)
    mask = ber < p

    return mask

BO_mask

BO_mask(X, p)

Black out missing mechanism. Removes values across dimensions.

Parameters

X : torch.FloatTensor, shape (n, d) Data for which missing values will be simulated. p : float Proportion of missing values to generate for variables which will have missing values.

Returns

mask : torch.BoolTensor Mask of generated missing values (True if the value is missing).

Source code in icu_benchmarks/imputation/amputations.py
def BO_mask(X, p):
    """
    Black out missing mechanism. Removes values across dimensions.

    Parameters
    ----------
    X : torch.FloatTensor, shape (n, d)
        Data for which missing values will be simulated.
    p : float
        Proportion of missing values to generate for variables which will have missing values.

    Returns
    -------
    mask : torch.BoolTensor
        Mask of generated missing values (True if the value is missing).
    """

    n, d = X.shape

    indices = torch.randperm(n)[: int(n * p)]
    mask = torch.zeros(n, d).bool()
    mask[indices, :] = True

    return mask

MAR_logistic_mask

MAR_logistic_mask(X, p, p_obs)

Missing at random mechanism with a logistic masking model. First, a subset of variables with no missing values is randomly selected. The remaining variables have missing values according to a logistic model with random weights, re-scaled so as to attain the desired proportion of missing values on those variables.

Parameters

X : torch.FloatTensor, shape (n, d) Data for which missing values will be simulated. p : float Proportion of missing values to generate for variables which will have missing values. p_obs : float Proportion of variables with no missing values that will be used for the logistic masking model.

Returns

mask : torch.BoolTensor Mask of generated missing values (True if the value is missing).

Source code in icu_benchmarks/imputation/amputations.py
def MAR_logistic_mask(X, p, p_obs):
    """
    Missing at random mechanism with a logistic masking model. First, a subset of variables with *no* missing values is
    randomly selected. The remaining variables have missing values according to a logistic model with random weights,
    re-scaled so as to attain the desired proportion of missing values on those variables.

    Parameters
    ----------
    X : torch.FloatTensor, shape (n, d)
        Data for which missing values will be simulated.
    p : float
        Proportion of missing values to generate for variables which will have missing values.
    p_obs : float
        Proportion of variables with *no* missing values that will be used for the logistic masking model.

    Returns
    -------
    mask : torch.BoolTensor
        Mask of generated missing values (True if the value is missing).
    """

    n, d = X.shape
    mask = torch.zeros(n, d).bool()

    # number of variables that will have no missing values (at least one variable)
    d_obs = max(int(p_obs * d), 1)
    # number of variables that will have missing values
    d_na = d - d_obs

    # Sample variables that will all be observed, and those with missing values
    idxs_obs = np.random.choice(d, d_obs, replace=False)
    idxs_nas = np.array([i for i in range(d) if i not in idxs_obs])

    # Other variables will have NA proportions that depend on those observed variables, through a logistic model
    # The parameters of this logistic model are random

    # Pick coefficients so that W^Tx has unit variance (avoids shrinking)
    coeffs = pick_coeffs(X, idxs_obs, idxs_nas)
    # Pick the intercepts to have a desired amount of missing values
    intercepts = fit_intercepts(X[:, idxs_obs], coeffs, p)

    ps = torch.sigmoid(X[:, idxs_obs].mm(coeffs) + intercepts)

    ber = torch.rand(n, d_na)
    mask[:, idxs_nas] = ber < ps

    return mask

MNAR_logistic_mask

MNAR_logistic_mask(X, p, p_params=0.3, exclude_inputs=True)

Missing not at random mechanism with a logistic masking model. It implements two mechanisms: (i) Missing probabilities are selected with a logistic model, taking all variables as inputs. Hence, values that are inputs can also be missing. (ii) Variables are split into a set of inputs for a logistic model, and a set whose missing probabilities are determined by the logistic model. Then inputs are then masked MCAR (hence, missing values from the second set will depend on masked values. In either case, weights are random and the intercept is selected to attain the desired proportion of missing values.

Parameters

X : torch.FloatTensor, shape (n, d) Data for which missing values will be simulated. p : float Proportion of missing values to generate for variables which will have missing values. p_params : float Proportion of variables that will be used for the logistic masking model (only if exclude_inputs). exclude_inputs : boolean, default=True True: mechanism (ii) is used, False: (i)

Returns

mask : torch.BoolTensor Mask of generated missing values (True if the value is missing).

Source code in icu_benchmarks/imputation/amputations.py
def MNAR_logistic_mask(X, p, p_params=0.3, exclude_inputs=True):
    """
    Missing not at random mechanism with a logistic masking model. It implements two mechanisms:
    (i) Missing probabilities are selected with a logistic model, taking all variables as inputs. Hence, values that are
    inputs can also be missing.
    (ii) Variables are split into a set of inputs for a logistic model, and a set whose missing probabilities are
    determined by the logistic model. Then inputs are then masked MCAR (hence, missing values from the second set will
    depend on masked values.
    In either case, weights are random and the intercept is selected to attain the desired proportion of missing values.

    Parameters
    ----------
    X : torch.FloatTensor, shape (n, d)
        Data for which missing values will be simulated.
    p : float
        Proportion of missing values to generate for variables which will have missing values.
    p_params : float
        Proportion of variables that will be used for the logistic masking model (only if exclude_inputs).
    exclude_inputs : boolean, default=True
        True: mechanism (ii) is used, False: (i)

    Returns
    -------
    mask : torch.BoolTensor
        Mask of generated missing values (True if the value is missing).
    """

    n, d = X.shape
    mask = torch.zeros(n, d).bool()

    # number of variables used as inputs (at least 1)
    d_params = max(int(p_params * d), 1) if exclude_inputs else d
    # number of variables masked with the logistic model
    d_na = d - d_params if exclude_inputs else d

    # Sample variables that will be parameters for the logistic regression:
    idxs_params = np.random.choice(d, d_params, replace=False) if exclude_inputs else np.arange(d)
    idxs_nas = np.array([i for i in range(d) if i not in idxs_params]) if exclude_inputs else np.arange(d)

    # Other variables will have NA proportions selected by a logistic model
    # The parameters of this logistic model are random.

    # Pick coefficients so that W^Tx has unit variance (avoids shrinking)
    coeffs = pick_coeffs(X, idxs_params, idxs_nas)
    # Pick the intercepts to have a desired amount of missing values
    intercepts = fit_intercepts(X[:, idxs_params], coeffs, p)

    ps = torch.sigmoid(X[:, idxs_params].mm(coeffs) + intercepts)

    ber = torch.rand(n, d_na)
    mask[:, idxs_nas] = ber < ps

    # If the inputs of the logistic model are excluded from MNAR missingness, mask some
    # values used in the logistic model at random
    # This makes the missingness of other variables potentially dependent on masked values

    if exclude_inputs:
        mask[:, idxs_params] = torch.rand(n, d_params) < p

    return mask

ampute_data

ampute_data(data, mechanism, p_miss, p_obs=0.3)

Generate missing values for specifics missing-data mechanism and proportion of missing values.

Parameters

data : DataFrame Data for which missing values will be simulated. mechanism : str, Indicates the missing-data mechanism to be used. ("MCAR", "MAR" or "MNAR") p_miss : float Proportion of missing values to generate for variables which will have missing values. p_obs : float If mecha = "MAR" or "MNAR", proportion of variables with no missing values that will be used for the logistic masking model.

Returns

imputed_data: DataFrame The data with the generated missing values.

Source code in icu_benchmarks/imputation/amputations.py
@gin.configurable("amputation")
def ampute_data(data, mechanism, p_miss, p_obs=0.3):
    """
    Generate missing values for specifics missing-data mechanism and proportion of missing values.

    Parameters
    ----------
    data : DataFrame
        Data for which missing values will be simulated.
    mechanism : str,
            Indicates the missing-data mechanism to be used. ("MCAR", "MAR" or "MNAR")
    p_miss : float
        Proportion of missing values to generate for variables which will have missing values.
    p_obs : float
            If mecha = "MAR" or "MNAR", proportion of variables with *no* missing values
            that will be used for the logistic masking model.

    Returns
    ----------
    imputed_data: DataFrame
        The data with the generated missing values.
    """
    logging.info(f"Applying {mechanism} amputation.")
    X = torch.tensor(data.values.astype(np.float32))

    if mechanism == "MAR":
        mask = MAR_logistic_mask(X, p_miss, p_obs)
    elif mechanism == "MNAR":
        mask = MNAR_logistic_mask(X, p_miss, p_obs)
    elif mechanism == "MCAR":
        mask = MCAR_mask(X, p_miss)
    elif mechanism == "BO":
        mask = BO_mask(X, p_miss)
    else:
        logging.error("Not a valid amputation mechanism. Missing-data mechanisms to be used are MAR, MCAR, MNAR or BO.")
        raise ValueError(f"Invalid amputation mechanism: {mechanism}")

    amputed_data = data.mask(mask)
    return amputed_data, mask

icu_benchmarks.imputation.baselines

Baseline imputation methods. These methods imported from other frameworks and are used as baselines for comparison.

Classes:

Name Description
KNNImputation

Imputation using Scikit-Learn K-Nearest Neighbour.

MICEImputation

Imputation using Scikit-Learn MICE.

MeanImputation

Mean imputation using Scikit-Learn SimpleImputer.

MedianImputation

Median imputation using Scikit-Learn SimpleImputer.

ZeroImputation

Zero imputation using Scikit-Learn SimpleImputer.

MostFrequentImputation

Most frequent imputation using Scikit-Learn SimpleImputer.

BRITSImputation

Bidirectional Recurrent Imputation for Time Series (BRITS) imputation using PyPots package.

SAITSImputation

Self-Attention based Imputation for Time Series (SAITS) imputation using PyPots package.

AttentionImputation

Attention based Imputation (Transformer) imputation using PyPots package.

Functions:

Name Description
import_optional

Import a package from the optional imputation extra (pip install "yaib[imputation]").

pypots_fit

PyPOTS models train on a dict holding the (stays, time steps, features) array with missing values.

pypots_impute

Imputes a batch with a fitted PyPOTS model, keeping the batch shape and device.

KNNImputation

KNNImputation(*args, n_neighbors=2, **kwargs)

Bases: ImputationWrapper

Imputation using Scikit-Learn K-Nearest Neighbour.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(self, *args, n_neighbors=2, **kwargs) -> None:
    super().__init__(*args, n_neighbors=n_neighbors, **kwargs)
    self.imputer = KNNImputer(n_neighbors=n_neighbors)

MICEImputation

MICEImputation(*args, max_iter=100, verbose=2, imputation_order='random', random_state=0, **kwargs)

Bases: ImputationWrapper

Imputation using Scikit-Learn MICE.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(
    self,
    *args,
    max_iter=100,
    verbose=2,
    imputation_order="random",
    random_state=0,
    **kwargs,
) -> None:
    super().__init__(
        *args,
        max_iter=max_iter,
        verbose=verbose,
        imputation_order=imputation_order,
        random_state=random_state,
        **kwargs,
    )
    self.imputer = IterativeImputer(
        estimator=LinearRegression(),
        max_iter=max_iter,
        verbose=verbose,
        imputation_order=imputation_order,
        random_state=random_state,
    )

MeanImputation

MeanImputation(*args, **kwargs)

Bases: ImputationWrapper

Mean imputation using Scikit-Learn SimpleImputer.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(self, *args, **kwargs) -> None:
    super().__init__(*args, **kwargs)
    self.imputer = SimpleImputer(strategy="mean")

MedianImputation

MedianImputation(*args, **kwargs)

Bases: ImputationWrapper

Median imputation using Scikit-Learn SimpleImputer.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(self, *args, **kwargs) -> None:
    super().__init__(*args, **kwargs)
    self.imputer = SimpleImputer(strategy="median")

ZeroImputation

ZeroImputation(*args, **kwargs)

Bases: ImputationWrapper

Zero imputation using Scikit-Learn SimpleImputer.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(self, *args, **kwargs) -> None:
    super().__init__(*args, **kwargs)
    self.imputer = SimpleImputer(strategy="constant", fill_value=0.0)

MostFrequentImputation

MostFrequentImputation(*args, **kwargs)

Bases: ImputationWrapper

Most frequent imputation using Scikit-Learn SimpleImputer.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(self, *args, **kwargs) -> None:
    super().__init__(*args, **kwargs)
    self.imputer = SimpleImputer(strategy="most_frequent")

BRITSImputation

BRITSImputation(*args, input_size, epochs=1, rnn_hidden_size=64, batch_size=256, **kwargs)

Bases: ImputationWrapper

Bidirectional Recurrent Imputation for Time Series (BRITS) imputation using PyPots package.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(self, *args, input_size, epochs=1, rnn_hidden_size=64, batch_size=256, **kwargs) -> None:
    super().__init__(
        *args,
        input_size=input_size,
        epochs=epochs,
        rnn_hidden_size=rnn_hidden_size,
        batch_size=batch_size,
        **kwargs,
    )
    BRITS = import_optional("pypots.imputation", "pypots").BRITS
    self.imputer = BRITS(
        n_steps=input_size[1],
        n_features=input_size[2],
        rnn_hidden_size=rnn_hidden_size,
        batch_size=batch_size,
        epochs=epochs,
        device="cuda" if torch.cuda.is_available() else "cpu",
    )

SAITSImputation

SAITSImputation(*args, input_size, epochs, n_layers, d_model, d_inner, n_head, d_k, d_v, dropout, **kwargs)

Bases: ImputationWrapper

Self-Attention based Imputation for Time Series (SAITS) imputation using PyPots package.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(
    self,
    *args,
    input_size,
    epochs,
    n_layers,
    d_model,
    d_inner,
    n_head,
    d_k,
    d_v,
    dropout,
    **kwargs,
) -> None:
    super().__init__(
        *args,
        input_size=input_size,
        epochs=epochs,
        n_layers=n_layers,
        d_model=d_model,
        d_inner=d_inner,
        n_head=n_head,
        d_k=d_k,
        d_v=d_v,
        dropout=dropout,
        **kwargs,
    )
    SAITS = import_optional("pypots.imputation", "pypots").SAITS
    self.imputer = SAITS(
        n_steps=input_size[1],
        n_features=input_size[2],
        n_layers=n_layers,
        d_model=d_model,
        d_ffn=d_inner,
        n_heads=n_head,
        d_k=d_k,
        d_v=d_v,
        dropout=dropout,
        epochs=epochs,
    )

AttentionImputation

AttentionImputation(*args, input_size, epochs, n_layers, d_model, d_inner, n_head, d_k, d_v, dropout, **kwargs)

Bases: ImputationWrapper

Attention based Imputation (Transformer) imputation using PyPots package.

Source code in icu_benchmarks/imputation/baselines.py
def __init__(
    self,
    *args,
    input_size,
    epochs,
    n_layers,
    d_model,
    d_inner,
    n_head,
    d_k,
    d_v,
    dropout,
    **kwargs,
) -> None:
    super().__init__(
        *args,
        input_size=input_size,
        epochs=epochs,
        n_layers=n_layers,
        d_model=d_model,
        d_inner=d_inner,
        n_head=n_head,
        d_k=d_k,
        d_v=d_v,
        dropout=dropout,
        **kwargs,
    )
    Transformer = import_optional("pypots.imputation", "pypots").Transformer
    self.imputer = Transformer(
        n_steps=input_size[1],
        n_features=input_size[2],
        n_layers=n_layers,
        d_model=d_model,
        d_ffn=d_inner,
        n_heads=n_head,
        d_k=d_k,
        d_v=d_v,
        dropout=dropout,
        epochs=epochs,
    )

import_optional

import_optional(module: str, package: str)

Import a package from the optional imputation extra (pip install "yaib[imputation]").

Source code in icu_benchmarks/imputation/baselines.py
def import_optional(module: str, package: str):
    """Import a package from the optional imputation extra (pip install "yaib[imputation]")."""
    try:
        return importlib.import_module(module)
    except ModuleNotFoundError as error:
        if (error.name or "").split(".")[0] != package:
            raise  # The package is installed, but one of its own dependencies is missing
        raise ModuleNotFoundError(
            f"{package} is required for this imputation model; install it with the imputation extra: "
            'pip install "yaib[imputation]" or uv sync --extra imputation',
            name=package,
        ) from error

pypots_fit

pypots_fit(imputer, train_dataset)

PyPOTS models train on a dict holding the (stays, time steps, features) array with missing values.

Source code in icu_benchmarks/imputation/baselines.py
def pypots_fit(imputer, train_dataset):
    """PyPOTS models train on a dict holding the (stays, time steps, features) array with missing values."""
    values = train_dataset.amputated_values.values.reshape(-1, train_dataset.maxlen, train_dataset.features_df.shape[1])
    imputer.fit({"X": values})

pypots_impute

pypots_impute(imputer, amputated_values)

Imputes a batch with a fitted PyPOTS model, keeping the batch shape and device.

Source code in icu_benchmarks/imputation/baselines.py
def pypots_impute(imputer, amputated_values):
    """Imputes a batch with a fitted PyPOTS model, keeping the batch shape and device."""
    values = amputated_values.reshape(-1, *amputated_values.shape[-2:]).to("cpu").numpy()
    return torch.Tensor(imputer.impute({"X": values})).reshape(amputated_values.shape).to(amputated_values.device)

icu_benchmarks.imputation.mlp

Classes:

Name Description
MLPImputation

Imputation model based on a Multi-Layer Perceptron (MLP).

MLPImputation

MLPImputation(*args, input_size, num_hidden_layers=3, hidden_layer_size=10, **kwargs)

Bases: ImputationWrapper

Imputation model based on a Multi-Layer Perceptron (MLP).

Source code in icu_benchmarks/imputation/mlp.py
def __init__(self, *args, input_size, num_hidden_layers=3, hidden_layer_size=10, **kwargs) -> None:
    super().__init__(
        *args,
        input_size=input_size,
        num_hidden_layers=num_hidden_layers,
        hidden_layer_size=hidden_layer_size,
        **kwargs,
    )
    self.model = [
        Flatten(),
        Linear(input_size[1] * input_size[2], hidden_layer_size),
        ReLU(),
        BatchNorm1d(hidden_layer_size),
    ]
    for _ in range(num_hidden_layers):
        self.model += [
            Linear(hidden_layer_size, hidden_layer_size),
            ReLU(),
            BatchNorm1d(hidden_layer_size),
        ]
    self.model += [
        Linear(hidden_layer_size, input_size[1] * input_size[2]),
        Sigmoid(),
    ]

    self.model = Sequential(*self.model)

icu_benchmarks.imputation.rnn

Classes:

Name Description
RNNImputation

Imputation model with Gated Recurrent Units (GRU) or Long-Short Term Memory Network (LSTM). Defaults to GRU.

BRNNImputation

Imputation model with Bidirectional Gated Recurrent Units (GRU) or Long-Short Term Memory Network (LSTM). Defaults to

RNNImputation

RNNImputation(*args, input_size, hidden_size=64, state_init='zero', cell='gru', **kwargs)

Bases: ImputationWrapper

Imputation model with Gated Recurrent Units (GRU) or Long-Short Term Memory Network (LSTM). Defaults to GRU.

Source code in icu_benchmarks/imputation/rnn.py
def __init__(self, *args, input_size, hidden_size=64, state_init="zero", cell="gru", **kwargs) -> None:
    super().__init__(
        *args,
        input_size=input_size,
        hidden_size=hidden_size,
        state_init=state_init,
        cell=cell,
        **kwargs,
    )
    self.input_size = input_size
    self.n_features = input_size[2]
    self.hidden_size = hidden_size
    self.state_init = state_init
    self.cell = cell

    if cell == "gru":
        cell = nn.GRUCell
    elif cell == "lstm":
        cell = nn.LSTMCell
    else:
        raise NotImplementedError(f'"{cell}" cell not implemented.')

    self.rnn = cell(self.n_features, self.hidden_size)
    self.fn = nn.Linear(self.hidden_size, self.n_features)

BRNNImputation

BRNNImputation(*args, input_size, hidden_size=64, state_init='zero', dropout=0.0, cell='gru', **kwargs)

Bases: ImputationWrapper

Imputation model with Bidirectional Gated Recurrent Units (GRU) or Long-Short Term Memory Network (LSTM). Defaults to GRU.

Source code in icu_benchmarks/imputation/rnn.py
def __init__(
    self,
    *args,
    input_size,
    hidden_size=64,
    state_init="zero",
    dropout=0.0,
    cell="gru",
    **kwargs,
) -> None:
    super().__init__(
        *args,
        input_size=input_size,
        hidden_size=hidden_size,
        state_init=state_init,
        dropout=dropout,
        cell=cell,
        **kwargs,
    )
    self.hidden_size = hidden_size
    self.fwd_rnn = RNNImputation(
        input_size=input_size,
        hidden_size=hidden_size,
        state_init=state_init,
        cell=cell,
    )
    self.bwd_rnn = RNNImputation(
        input_size=input_size,
        hidden_size=hidden_size,
        state_init=state_init,
        cell=cell,
    )
    self.dropout = nn.Dropout(dropout)
    self.fn = nn.Linear(2 * hidden_size, input_size[2])

icu_benchmarks.imputation.np

Classes:

Name Description
NPImputation

Imputation using Neural Processes. Implementation adapted from https://github.com/EmilienDupont/neural-processes/.

NeuralProcess

Class that implements neural processes.

NPImputation

NPImputation(input_size, encoder_layers, encoder_h_dim, decoder_layers, decoder_h_dim, r_dim, z_dim, train_sample_times, val_sample_times, test_sample_times, predict_sample_times, *args, **kwargs)

Bases: ImputationWrapper

Imputation using Neural Processes. Implementation adapted from https://github.com/EmilienDupont/neural-processes/. Provides imputation wrapper for NeuralProcess class.

Source code in icu_benchmarks/imputation/np.py
def __init__(
    self,
    input_size,
    encoder_layers,
    encoder_h_dim,
    decoder_layers,
    decoder_h_dim,
    r_dim,
    z_dim,
    train_sample_times,
    val_sample_times,
    test_sample_times,
    predict_sample_times,
    *args,
    **kwargs,
) -> None:
    super().__init__(
        input_size=input_size,
        encoder_layers=encoder_layers,
        encoder_h_dim=encoder_h_dim,
        decoder_layers=decoder_layers,
        decoder_h_dim=decoder_h_dim,
        r_dim=r_dim,
        z_dim=z_dim,
        train_sample_times=train_sample_times,
        val_sample_times=val_sample_times,
        test_sample_times=test_sample_times,
        predict_sample_times=predict_sample_times,
        *args,
        **kwargs,
    )

    self.x_dim = input_size[2]
    self.y_dim = input_size[2]
    self.z_dim = z_dim

    self.train_sample_times = train_sample_times
    self.val_sample_times = val_sample_times
    self.test_sample_times = test_sample_times
    self.predict_sample_times = predict_sample_times

    self.model = NeuralProcess(
        self.x_dim,
        self.y_dim,
        encoder_layers,
        encoder_h_dim,
        decoder_layers,
        decoder_h_dim,
        r_dim,
        z_dim,
    )

NeuralProcess

NeuralProcess(x_dim, y_dim, encoder_layers, encoder_h_dim, decoder_layers, decoder_h_dim, r_dim, z_dim)

Bases: Module

Class that implements neural processes.

Source code in icu_benchmarks/imputation/np.py
def __init__(
    self,
    x_dim,
    y_dim,
    encoder_layers,
    encoder_h_dim,
    decoder_layers,
    decoder_h_dim,
    r_dim,
    z_dim,
):
    super().__init__()

    self.x_dim = x_dim
    self.y_dim = y_dim
    self.r_dim = r_dim
    self.z_dim = z_dim

    # Initialize encoders/decoder
    self.encoder = MLPEncoder(x_dim, y_dim, encoder_h_dim, encoder_layers, r_dim)

    self.latent_encoder = MuEncoder(r_dim=r_dim, z_dim=z_dim)

    self.decoder = Decoder(decoder_h_dim, decoder_layers, x_dim, y_dim, z_dim)

icu_benchmarks.imputation.csdi

Classes:

Name Description
CSDI

Conditional Score-based Diffusion Models for Imputation (CSDI) of Time Series. See https://arxiv.org/abs/2107.03502

CSDI

CSDI(input_size, time_step_embedding_size, feature_embedding_size, unconditional, target_strategy, num_diffusion_steps, diffusion_step_embedding_dim, n_attention_heads, num_residual_layers, noise_schedule, beta_start, beta_end, n_samples, conv_channels, *args, **kwargs)

Bases: ImputationWrapper

Conditional Score-based Diffusion Models for Imputation (CSDI) of Time Series. See https://arxiv.org/abs/2107.03502 for details.

Source code in icu_benchmarks/imputation/csdi.py
def __init__(
    self,
    input_size,
    time_step_embedding_size,
    feature_embedding_size,
    unconditional,
    target_strategy,
    num_diffusion_steps,
    diffusion_step_embedding_dim,
    n_attention_heads,
    num_residual_layers,
    noise_schedule,
    beta_start,
    beta_end,
    n_samples,
    conv_channels,
    *args,
    **kwargs,
):
    super().__init__(
        input_size=input_size,
        time_step_embedding_size=time_step_embedding_size,
        feature_embedding_size=feature_embedding_size,
        unconditional=unconditional,
        target_strategy=target_strategy,
        num_diffusion_steps=num_diffusion_steps,
        diffusion_step_embedding_dim=diffusion_step_embedding_dim,
        n_attention_heads=n_attention_heads,
        num_residual_layers=num_residual_layers,
        noise_schedule=noise_schedule,
        beta_start=beta_start,
        beta_end=beta_end,
        n_samples=n_samples,
        conv_channels=conv_channels,
        *args,
        **kwargs,
    )
    self.target_dim = input_size[2]
    self.n_samples = n_samples

    self.emb_time_dim = time_step_embedding_size
    self.emb_feature_dim = feature_embedding_size
    self.is_unconditional = unconditional
    self.target_strategy = target_strategy

    self.emb_total_dim = self.emb_time_dim + self.emb_feature_dim
    if not self.is_unconditional:
        self.emb_total_dim += 1  # for conditional mask
    self.embed_layer = nn.Embedding(num_embeddings=self.target_dim, embedding_dim=self.emb_feature_dim)

    input_dim = 1 if self.is_unconditional else 2
    self.diffmodel = diff_CSDI(
        conv_channels,
        num_diffusion_steps,
        diffusion_step_embedding_dim,
        self.emb_total_dim,
        n_attention_heads,
        num_residual_layers,
        input_dim,
    )

    # parameters for diffusion models
    self.num_steps = num_diffusion_steps
    if noise_schedule == "quad":
        self.beta = np.linspace(beta_start**0.5, beta_end**0.5, self.num_steps) ** 2
    elif noise_schedule == "linear":
        self.beta = np.linspace(beta_start, beta_end, self.num_steps)

    self.alpha_hat = 1 - self.beta
    self.alpha = np.cumprod(self.alpha_hat)
    self.alpha_torch = torch.tensor(self.alpha).float().unsqueeze(1).unsqueeze(1)

icu_benchmarks.imputation.diffusion

Classes:

Name Description
SimpleDiffusionModel

Simple Diffusion Model for Imputation. See https://arxiv.org/abs/2006.11239 for more details.

SimpleDiffusionModel

SimpleDiffusionModel(input_size, n_onedirectional_conv, T, min_noise, max_noise, noise_scheduler, *args, **kwargs)

Bases: ImputationWrapper

Simple Diffusion Model for Imputation. See https://arxiv.org/abs/2006.11239 for more details.

Methods:

Name Description
predict_step

Impute the missing values by running the reverse diffusion from pure noise. At each timestep, the

backward_diffusion_sample

Function that takes a noised image at some timestep t and the noise prediction and tries to compute the original

Source code in icu_benchmarks/imputation/diffusion.py
def __init__(
    self,
    input_size,
    n_onedirectional_conv,
    T,
    min_noise,
    max_noise,
    noise_scheduler,
    *args,
    **kwargs,
):
    super().__init__(
        n_onedirectional_conv=n_onedirectional_conv,
        T=T,
        min_noise=min_noise,
        max_noise=max_noise,
        noise_scheduler=noise_scheduler,
        *args,
        **kwargs,
    )

    self.n_onedirectional_conv = n_onedirectional_conv
    self.T = T
    self.min_noise = min_noise
    self.max_noise = max_noise
    self.noise_scheduler = noise_scheduler

    # == Noise Schedulers == #
    # Linear
    if self.noise_scheduler == "linear":
        betas = torch.linspace(self.min_noise, self.max_noise, self.T)
    # Quadratic
    elif self.noise_scheduler == "quadratic":
        betas = torch.linspace(self.min_noise**0.5, self.max_noise**0.5, self.T) ** 2
    # Cosine
    elif self.noise_scheduler == "cosine":
        x = torch.linspace(0, self.T, self.T + 1)
        alphas_cumprod = torch.cos(((x / self.T) + 0.008) / (1 + 0.008) * torch.pi * 0.5) ** 2
        alphas_cumprod = alphas_cumprod / alphas_cumprod[0]
        betas = 1 - (alphas_cumprod[1:] / alphas_cumprod[:-1])
        betas = torch.clip(betas, 0.0001, 0.9999)
    # Sigmoid
    elif self.noise_scheduler == "sigmoid":
        betas = torch.linspace(-6, 6, self.T)
        betas = torch.sigmoid(betas) * (self.max_noise - self.min_noise) + self.min_noise
    # Error
    else:
        raise NotImplementedError(
            "Noise Scheduler must be linear, quadratic, cosine or sigmoid.\n Your Entry: [%s] is not implemented"
            % self.noise_scheduler
        )

    # Helper Values. As buffers they move with the model, also when a restored model is moved to a GPU for test
    # or predict. They follow from the noise schedule, so they are not saved in checkpoints.
    alphas = 1.0 - betas
    alphas_cumprod = torch.cumprod(alphas, dim=0)
    alphas_cumprod_prev = F.pad(alphas_cumprod[:-1], (1, 0), value=1.0)
    schedule = {
        "betas": betas,
        "alphas": alphas,
        "alphas_cumprod": alphas_cumprod,
        "alphas_cumprod_prev": alphas_cumprod_prev,
        "sqrt_recip_alphas": torch.sqrt(1.0 / alphas),
        "sqrt_alphas_cumprod": torch.sqrt(alphas_cumprod),
        "sqrt_one_minus_alphas_cumprod": torch.sqrt(1.0 - alphas_cumprod),
        "posterior_variance": betas * (1.0 - alphas_cumprod_prev) / (1.0 - alphas_cumprod),
    }
    for name, values in schedule.items():
        self.register_buffer(name, values, persistent=False)

    # Store Input Size
    self.input_size = input_size

    # Time embedding
    self.time_mlp = nn.Sequential(
        SinusoidalPositionEmbeddings(input_size[2]),
        nn.Linear(input_size[2], input_size[2]),
        nn.ReLU(),
    )

    # Blocks
    self.downs = nn.ModuleList()
    self.ups = nn.ModuleList()

    for i in range(self.n_onedirectional_conv):
        self.downs.append(Block(input_size, i))
        self.ups.append(Block(input_size, (self.n_onedirectional_conv - i), up=True))

predict_step

predict_step(data, amputation_mask)

Impute the missing values by running the reverse diffusion from pure noise. At each timestep, the observed values are replaced by their forward diffusion sample for that timestep.

Source code in icu_benchmarks/imputation/diffusion.py
def predict_step(self, data, amputation_mask):
    """Impute the missing values by running the reverse diffusion from pure noise. At each timestep, the
    observed values are replaced by their forward diffusion sample for that timestep."""
    amputated = torch.nan_to_num(data, nan=0.0)
    amputation_mask = amputation_mask.bool()

    x = torch.randn_like(amputated)
    for t in range(self.T - 1, -1, -1):
        t_batch = torch.full((amputated.shape[0],), t, device=amputated.device, dtype=torch.long)
        observed_t, _ = self.forward_diffusion_sample(amputated, t_batch)
        x = torch.where(amputation_mask, x, observed_t)
        x = self.backward_diffusion_sample(self(x, t_batch), x, t_batch, t_index=t)

    return torch.where(amputation_mask, x, amputated)

backward_diffusion_sample

backward_diffusion_sample(noise_pred, x_t, t, t_index=0)

Function that takes a noised image at some timestep t and the noise prediction and tries to compute the original sample. The t needs to be one specific timestamp -> always the same value. It does not have to be like this in the forward diffusion sample function

Source code in icu_benchmarks/imputation/diffusion.py
def backward_diffusion_sample(self, noise_pred, x_t, t, t_index=0):
    """Function that takes a noised image at some timestep t and the noise prediction and tries to compute the original
    sample. The t needs to be one specific timestamp -> always the same value. It does not have to be like
    this in the forward diffusion sample function"""

    betas_t = self.get_index_from_list(self.betas, t, x_t.shape)
    sqrt_one_minus_alphas_cumprod_t = self.get_index_from_list(self.sqrt_one_minus_alphas_cumprod, t, x_t.shape)
    sqrt_recip_alphas_t = self.get_index_from_list(self.sqrt_recip_alphas, t, x_t.shape)
    posterior_variance_t = self.get_index_from_list(self.posterior_variance, t, x_t.shape)

    model_mean = sqrt_recip_alphas_t * (x_t - betas_t * noise_pred / sqrt_one_minus_alphas_cumprod_t)

    if t_index == 0:
        return model_mean
    else:
        noise = torch.randn_like(x_t)
        return model_mean + torch.sqrt(posterior_variance_t) * noise

icu_benchmarks.imputation.simple_diffusion

Classes:

Name Description
SimpleDiffusionModel

Imputation model based on a Simple Diffusion Model.

SimpleDiffusionModel

SimpleDiffusionModel(*args, input_size, **kwargs)

Bases: ImputationWrapper

Imputation model based on a Simple Diffusion Model. Adapted from https://colab.research.google.com/drive/1sjy9odlSSy0RBVgMTgP7s99NXsqglsUL.

Methods:

Name Description
get_index_from_list

Returns a specific index t of a passed list of values vals

forward_diffusion_sample

Takes an image and a timestep as input and

predict_step

Impute the missing values by running the reverse diffusion from pure noise. At each timestep, the

Source code in icu_benchmarks/imputation/simple_diffusion.py
def __init__(self, *args, input_size, **kwargs):
    super().__init__(*args, input_size=input_size, **kwargs)

    down_channels = (25, 20, 18, 15)
    up_channels = (15, 18, 20, 25)
    time_emb_dim = 6

    self.input_size = input_size

    # Time embedding
    self.time_mlp = nn.Sequential(
        SinusoidalPositionEmbeddings(time_emb_dim),
        nn.Linear(time_emb_dim, time_emb_dim),
        nn.ReLU(),
    )

    # Initial projection
    self.conv0 = nn.Conv1d(input_size[1], down_channels[0], 2)

    # Downsample
    self.downs = nn.ModuleList(
        [Block(down_channels[i], down_channels[i + 1], time_emb_dim) for i in range(len(down_channels) - 1)]
    )

    # Upsample
    self.ups = nn.ModuleList(
        [Block(up_channels[i], up_channels[i + 1], time_emb_dim, up=True) for i in range(len(up_channels) - 1)]
    )

    # Final Output
    self.output = nn.ConvTranspose1d(up_channels[-1], input_size[1], 2)

    # Pre-calculate the terms of the closed-form diffusion. As buffers they move with the model, also when a
    # restored model is moved to a GPU for test or predict (on_fit_start does not run there). They follow from
    # T, so they are not saved in checkpoints.
    betas = self.linear_beta_schedule(timesteps=self.T)
    alphas = 1.0 - betas
    alphas_cumprod = torch.cumprod(alphas, dim=0)
    alphas_cumprod_prev = F.pad(alphas_cumprod[:-1], (1, 0), value=1.0)
    schedule = {
        "betas": betas,
        "alphas": alphas,
        "alphas_cumprod": alphas_cumprod,
        "alphas_cumprod_prev": alphas_cumprod_prev,
        "sqrt_recip_alphas": torch.sqrt(1.0 / alphas),
        "sqrt_alphas_cumprod": torch.sqrt(alphas_cumprod),
        "sqrt_one_minus_alphas_cumprod": torch.sqrt(1.0 - alphas_cumprod),
        "posterior_variance": betas * (1.0 - alphas_cumprod_prev) / (1.0 - alphas_cumprod),
    }
    for name, values in schedule.items():
        self.register_buffer(name, values, persistent=False)

get_index_from_list

get_index_from_list(vals, t, x_shape)

Returns a specific index t of a passed list of values vals while considering the batch dimension.

Source code in icu_benchmarks/imputation/simple_diffusion.py
def get_index_from_list(self, vals, t, x_shape):
    """
    Returns a specific index t of a passed list of values vals
    while considering the batch dimension.
    """
    batch_size = t.shape[0]
    out = vals.gather(-1, t)
    return out.reshape(batch_size, *((1,) * (len(x_shape) - 1))).to(t.device)

forward_diffusion_sample

forward_diffusion_sample(x_0, t)

Takes an image and a timestep as input and returns the noisy version of it

Source code in icu_benchmarks/imputation/simple_diffusion.py
def forward_diffusion_sample(self, x_0, t):
    """
    Takes an image and a timestep as input and
    returns the noisy version of it
    """
    noise = torch.randn_like(x_0)
    sqrt_alphas_cumprod_t = self.get_index_from_list(self.sqrt_alphas_cumprod, t, x_0.shape)
    sqrt_one_minus_alphas_cumprod_t = self.get_index_from_list(self.sqrt_one_minus_alphas_cumprod, t, x_0.shape)
    # mean + variance
    return (
        sqrt_alphas_cumprod_t * x_0 + sqrt_one_minus_alphas_cumprod_t * noise,
        noise,
    )

predict_step

predict_step(data, amputation_mask)

Impute the missing values by running the reverse diffusion from pure noise. At each timestep, the observed values are replaced by their forward diffusion sample for that timestep.

Source code in icu_benchmarks/imputation/simple_diffusion.py
def predict_step(self, data, amputation_mask):
    """Impute the missing values by running the reverse diffusion from pure noise. At each timestep, the
    observed values are replaced by their forward diffusion sample for that timestep."""
    amputated = torch.nan_to_num(data, nan=0.0)
    amputation_mask = amputation_mask.bool()

    x = torch.randn_like(amputated)
    for t in range(self.T - 1, -1, -1):
        t_batch = torch.full((amputated.shape[0],), t, device=self.device, dtype=torch.long)
        observed_t, _ = self.forward_diffusion_sample(amputated, t_batch)
        x = torch.where(amputation_mask, x, observed_t)

        betas_t = self.get_index_from_list(self.betas, t_batch, x.shape)
        sqrt_one_minus_alphas_cumprod_t = self.get_index_from_list(self.sqrt_one_minus_alphas_cumprod, t_batch, x.shape)
        sqrt_recip_alphas_t = self.get_index_from_list(self.sqrt_recip_alphas, t_batch, x.shape)
        x = sqrt_recip_alphas_t * (x - betas_t * self(x, t_batch) / sqrt_one_minus_alphas_cumprod_t)
        if t > 0:
            posterior_variance_t = self.get_index_from_list(self.posterior_variance, t_batch, x.shape)
            x = x + torch.sqrt(posterior_variance_t) * torch.randn_like(x)

    return torch.where(amputation_mask, x, amputated)

icu_benchmarks.imputation.diffwave

Classes:

Name Description
DiffWaveImputer

Imputation model based on DiffWave (https://arxiv.org/abs/2009.09761). Adapted from

Functions:

Name Description
calc_diffusion_step_embedding

Embed a diffusion step $t$ into a higher dimensional space

calc_diffusion_hyperparams

Compute diffusion process hyperparameters

std_normal

Generate the standard Gaussian variable of a certain size

DiffWaveImputer

DiffWaveImputer(in_channels, res_channels, skip_channels, out_channels, num_res_layers, dilation_cycle, diffusion_step_embed_dim_in, diffusion_step_embed_dim_mid, diffusion_step_embed_dim_out, diffusion_time_steps, beta_0, beta_T, *args, **kwargs)

Bases: ImputationWrapper

Imputation model based on DiffWave (https://arxiv.org/abs/2009.09761). Adapted from https://github.com/AI4HealthUOL/SSSD/blob/main/src/imputers/DiffWaveImputer.py

Methods:

Name Description
sampling

Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{ heta}(x_{t-1}|x_t)

Source code in icu_benchmarks/imputation/diffwave.py
def __init__(
    self,
    in_channels,
    res_channels,
    skip_channels,
    out_channels,
    num_res_layers,
    dilation_cycle,
    diffusion_step_embed_dim_in,
    diffusion_step_embed_dim_mid,
    diffusion_step_embed_dim_out,
    diffusion_time_steps,
    beta_0,
    beta_T,
    *args,
    **kwargs,
):
    super(DiffWaveImputer, self).__init__(
        in_channels=in_channels,
        res_channels=res_channels,
        skip_channels=skip_channels,
        out_channels=out_channels,
        num_res_layers=num_res_layers,
        dilation_cycle=dilation_cycle,
        diffusion_step_embed_dim_in=diffusion_step_embed_dim_in,
        diffusion_step_embed_dim_mid=diffusion_step_embed_dim_mid,
        diffusion_step_embed_dim_out=diffusion_step_embed_dim_out,
        diffusion_time_steps=diffusion_time_steps,
        beta_0=beta_0,
        beta_T=beta_T,
        *args,
        **kwargs,
    )

    self.init_conv = nn.Sequential(Conv(in_channels, res_channels, kernel_size=1), nn.ReLU())

    self.residual_layer = Residual_group(
        res_channels=res_channels,
        skip_channels=skip_channels,
        num_res_layers=num_res_layers,
        dilation_cycle=dilation_cycle,
        diffusion_step_embed_dim_in=diffusion_step_embed_dim_in,
        diffusion_step_embed_dim_mid=diffusion_step_embed_dim_mid,
        diffusion_step_embed_dim_out=diffusion_step_embed_dim_out,
        in_channels=in_channels,
    )

    self.final_conv = nn.Sequential(
        Conv(skip_channels, skip_channels, kernel_size=1),
        nn.ReLU(),
        ZeroConv1d(skip_channels, out_channels),
    )

    self.diffusion_parameters = calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T)

sampling

sampling(cond, mask)

Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{ heta}(x_{t-1}|x_t)

Parameters: net (torch network): the wavenet model size (tuple): size of tensor to be generated, usually is (number of audios to generate, channels=1, length of audio) diffusion_hyperparams (dict): dictionary of diffusion hyperparameters returned by calc_diffusion_hyperparams note, the tensors need to be cuda tensors

Returns: the generated audio(s) in torch.tensor, shape=size

Source code in icu_benchmarks/imputation/diffwave.py
def sampling(self, cond, mask):
    """
    Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{\theta}(x_{t-1}|x_t)

    Parameters:
    net (torch network):            the wavenet model
    size (tuple):                   size of tensor to be generated,
                                    usually is (number of audios to generate, channels=1, length of audio)
    diffusion_hyperparams (dict):   dictionary of diffusion hyperparameters returned by calc_diffusion_hyperparams
                                    note, the tensors need to be cuda tensors

    Returns:
    the generated audio(s) in torch.tensor, shape=size
    """

    Alpha, Alpha_bar, Sigma = (
        self.diffusion_parameters["Alpha"],
        self.diffusion_parameters["Alpha_bar"],
        self.diffusion_parameters["Sigma"],
    )

    T = self.hparams.diffusion_time_steps
    assert len(Alpha) == T
    assert len(Alpha_bar) == T
    assert len(Sigma) == T

    B, _, _ = cond.shape
    x = std_normal(cond.shape, self.device)

    for t in range(T - 1, -1, -1):
        x = x * (1 - mask).float() + cond * mask.float()
        diffusion_steps = (t * torch.ones((B, 1))).to(self.device)  # use the corresponding reverse step
        epsilon_theta = self(
            (
                x,
                cond,
                mask,
                diffusion_steps,
            )
        )  # predict \epsilon according to \epsilon_\theta
        # update x_{t-1} to \mu_\theta(x_t)
        x = (x - (1 - Alpha[t]) / torch.sqrt(1 - Alpha_bar[t]) * epsilon_theta) / torch.sqrt(Alpha[t])
        if t > 0:
            x = x + Sigma[t] * std_normal(cond.shape, self.device)  # add the variance term to x_{t-1}

    return x

calc_diffusion_step_embedding

calc_diffusion_step_embedding(diffusion_steps, diffusion_step_embed_dim_in, device)

Embed a diffusion step $t$ into a higher dimensional space E.g. the embedding vector in the 128-dimensional space is [sin(t * 10^(04/63)), ... , sin(t * 10^(634/63)), cos(t * 10^(04/63)), ... , cos(t * 10^(634/63))] Parameters: diffusion_steps (torch.long tensor, shape=(batchsize, 1)): diffusion steps for batch data diffusion_step_embed_dim_in (int, default=128): dimensionality of the embedding space for discrete diffusion steps

Returns: the embedding vectors (torch.tensor, shape=(batchsize, diffusion_step_embed_dim_in)):

Source code in icu_benchmarks/imputation/diffwave.py
def calc_diffusion_step_embedding(diffusion_steps, diffusion_step_embed_dim_in, device):
    """
    Embed a diffusion step $t$ into a higher dimensional space
    E.g. the embedding vector in the 128-dimensional space is
    [sin(t * 10^(0*4/63)), ... , sin(t * 10^(63*4/63)), cos(t * 10^(0*4/63)), ... , cos(t * 10^(63*4/63))]
    Parameters:
    diffusion_steps (torch.long tensor, shape=(batchsize, 1)):
                                diffusion steps for batch data
    diffusion_step_embed_dim_in (int, default=128):
                                dimensionality of the embedding space for discrete diffusion steps

    Returns:
    the embedding vectors (torch.tensor, shape=(batchsize, diffusion_step_embed_dim_in)):
    """

    assert diffusion_step_embed_dim_in % 2 == 0

    half_dim = diffusion_step_embed_dim_in // 2
    _embed = np.log(10000) / (half_dim - 1)
    _embed = torch.exp(torch.arange(half_dim) * -_embed).to(device)
    _embed = diffusion_steps * _embed
    diffusion_step_embed = torch.cat((torch.sin(_embed), torch.cos(_embed)), 1)

    return diffusion_step_embed

calc_diffusion_hyperparams

calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T)

Compute diffusion process hyperparameters

Parameters: T (int): number of diffusion steps beta_0 and beta_T (float): beta schedule start/end value, where any beta_t in the middle is linearly interpolated

a dictionary of diffusion hyperparameters including: T (int), Beta/Alpha/Alpha_bar/Sigma (torch.tensor on cpu, shape=(T, )) These cpu tensors are changed to cuda tensors on each individual gpu

Source code in icu_benchmarks/imputation/diffwave.py
def calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T):
    """
    Compute diffusion process hyperparameters

    Parameters:
    T (int):                    number of diffusion steps
    beta_0 and beta_T (float):  beta schedule start/end value,
                                where any beta_t in the middle is linearly interpolated

    Returns:
    a dictionary of diffusion hyperparameters including:
        T (int), Beta/Alpha/Alpha_bar/Sigma (torch.tensor on cpu, shape=(T, ))
        These cpu tensors are changed to cuda tensors on each individual gpu
    """

    Beta = torch.linspace(beta_0, beta_T, diffusion_time_steps)  # Linear schedule
    Alpha = 1 - Beta
    Alpha_bar = Alpha + 0
    Beta_tilde = Beta + 0
    for t in range(1, diffusion_time_steps):
        Alpha_bar[t] *= Alpha_bar[t - 1]  # \bar{\alpha}_t = \prod_{s=1}^t \alpha_s
        Beta_tilde[t] *= (1 - Alpha_bar[t - 1]) / (1 - Alpha_bar[t])  # \tilde{\beta}_t = \beta_t * (1-\bar{\alpha}_{t-1})
        # / (1-\bar{\alpha}_t)
    Sigma = torch.sqrt(Beta_tilde)  # \sigma_t^2  = \tilde{\beta}_t

    _dh = {}
    _dh["T"], _dh["Beta"], _dh["Alpha"], _dh["Alpha_bar"], _dh["Sigma"] = (
        diffusion_time_steps,
        Beta,
        Alpha,
        Alpha_bar,
        Sigma,
    )
    diffusion_hyperparams = _dh
    return diffusion_hyperparams

std_normal

std_normal(size, device)

Generate the standard Gaussian variable of a certain size

Source code in icu_benchmarks/imputation/diffwave.py
def std_normal(size, device):
    """
    Generate the standard Gaussian variable of a certain size
    """

    return torch.normal(0, 1, size=size).to(device)

icu_benchmarks.imputation.sssds4

Classes:

Name Description
SSSDS4

Implements the SSSD model from https://arxiv.org/abs/2208.09399.

Functions:

Name Description
std_normal

Generate the standard Gaussian variable of a certain size

calc_diffusion_step_embedding

Embed a diffusion step $t$ into a higher dimensional space

calc_diffusion_hyperparams

Compute diffusion process hyperparameters

SSSDS4

SSSDS4(input_size, res_channels, skip_channels, num_res_layers, diffusion_step_embed_dim_in, diffusion_step_embed_dim_mid, diffusion_step_embed_dim_out, s4_lmax, s4_d_state, s4_dropout, s4_bidirectional, s4_layernorm, diffusion_time_steps, beta_0, beta_T, *args, **kwargs: str)

Bases: ImputationWrapper

Implements the SSSD model from https://arxiv.org/abs/2208.09399.

Methods:

Name Description
sampling

Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{ heta}(x_{t-1}|x_t)

Source code in icu_benchmarks/imputation/sssds4.py
def __init__(
    self,
    input_size,
    res_channels,
    skip_channels,
    num_res_layers,
    diffusion_step_embed_dim_in,
    diffusion_step_embed_dim_mid,
    diffusion_step_embed_dim_out,
    s4_lmax,
    s4_d_state,
    s4_dropout,
    s4_bidirectional,
    s4_layernorm,
    diffusion_time_steps,
    beta_0,
    beta_T,
    *args,
    **kwargs: str,
):
    super(SSSDS4, self).__init__(
        input_size=input_size,
        res_channels=res_channels,
        skip_channels=skip_channels,
        num_res_layers=num_res_layers,
        diffusion_step_embed_dim_in=diffusion_step_embed_dim_in,
        diffusion_step_embed_dim_mid=diffusion_step_embed_dim_mid,
        diffusion_step_embed_dim_out=diffusion_step_embed_dim_out,
        s4_lmax=s4_lmax,
        s4_d_state=s4_d_state,
        s4_dropout=s4_dropout,
        s4_bidirectional=s4_bidirectional,
        s4_layernorm=s4_layernorm,
        diffusion_time_steps=diffusion_time_steps,
        beta_0=beta_0,
        beta_T=beta_T,
        *args,
        **kwargs,
    )

    num_channels = input_size[2]
    self.init_conv = nn.Sequential(Conv(num_channels, res_channels, kernel_size=1), nn.ReLU())

    self.residual_layer = Residual_group(
        res_channels=res_channels,
        skip_channels=skip_channels,
        num_res_layers=num_res_layers,
        diffusion_step_embed_dim_in=diffusion_step_embed_dim_in,
        diffusion_step_embed_dim_mid=diffusion_step_embed_dim_mid,
        diffusion_step_embed_dim_out=diffusion_step_embed_dim_out,
        in_channels=num_channels,
        s4_lmax=s4_lmax,
        s4_d_state=s4_d_state,
        s4_dropout=s4_dropout,
        s4_bidirectional=s4_bidirectional,
        s4_layernorm=s4_layernorm,
    )

    self.final_conv = nn.Sequential(
        Conv(skip_channels, skip_channels, kernel_size=1),
        nn.ReLU(),
        ZeroConv1d(skip_channels, num_channels),
    )

    self.diffusion_parameters = calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T)

sampling

sampling(cond, mask)

Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{ heta}(x_{t-1}|x_t)

Parameters: net (torch network): the wavenet model size (tuple): size of tensor to be generated, usually is (number of audios to generate, channels=1, length of audio) diffusion_hyperparams (dict): dictionary of diffusion hyperparameters returned by calc_diffusion_hyperparams note, the tensors need to be cuda tensors

Returns: the generated audio(s) in torch.tensor, shape=size

Source code in icu_benchmarks/imputation/sssds4.py
def sampling(self, cond, mask):
    """
    Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{\theta}(x_{t-1}|x_t)

    Parameters:
    net (torch network):            the wavenet model
    size (tuple):                   size of tensor to be generated,
                                    usually is (number of audios to generate, channels=1, length of audio)
    diffusion_hyperparams (dict):   dictionary of diffusion hyperparameters returned by calc_diffusion_hyperparams
                                    note, the tensors need to be cuda tensors

    Returns:
    the generated audio(s) in torch.tensor, shape=size
    """

    Alpha, Alpha_bar, Sigma = (
        self.diffusion_parameters["Alpha"],
        self.diffusion_parameters["Alpha_bar"],
        self.diffusion_parameters["Sigma"],
    )

    T = self.hparams.diffusion_time_steps
    assert len(Alpha) == T
    assert len(Alpha_bar) == T
    assert len(Sigma) == T

    B, _, _ = cond.shape
    x = std_normal(cond.shape, self.device)

    for t in range(T - 1, -1, -1):
        x = x * (1 - mask).float() + cond * mask.float()
        diffusion_steps = (t * torch.ones((B, 1))).to(self.device)  # use the corresponding reverse step
        epsilon_theta = self(
            (
                x,
                cond,
                mask,
                diffusion_steps,
            )
        )  # predict \epsilon according to \epsilon_\theta
        # update x_{t-1} to \mu_\theta(x_t)
        x = (x - (1 - Alpha[t]) / torch.sqrt(1 - Alpha_bar[t]) * epsilon_theta) / torch.sqrt(Alpha[t])
        if t > 0:
            x = x + Sigma[t] * std_normal(cond.shape, self.device)  # add the variance term to x_{t-1}

    return x

std_normal

std_normal(size, device)

Generate the standard Gaussian variable of a certain size

Source code in icu_benchmarks/imputation/sssds4.py
def std_normal(size, device):
    """
    Generate the standard Gaussian variable of a certain size
    """

    return torch.normal(0, 1, size=size).to(device)

calc_diffusion_step_embedding

calc_diffusion_step_embedding(diffusion_steps, diffusion_step_embed_dim_in, device)

Embed a diffusion step $t$ into a higher dimensional space E.g. the embedding vector in the 128-dimensional space is [sin(t * 10^(04/63)), ... , sin(t * 10^(634/63)), cos(t * 10^(04/63)), ... , cos(t * 10^(634/63))] Parameters: diffusion_steps (torch.long tensor, shape=(batchsize, 1)): diffusion steps for batch data diffusion_step_embed_dim_in (int, default=128): dimensionality of the embedding space for discrete diffusion steps

Returns: the embedding vectors (torch.tensor, shape=(batchsize, diffusion_step_embed_dim_in)):

Source code in icu_benchmarks/imputation/sssds4.py
def calc_diffusion_step_embedding(diffusion_steps, diffusion_step_embed_dim_in, device):
    """
    Embed a diffusion step $t$ into a higher dimensional space
    E.g. the embedding vector in the 128-dimensional space is
    [sin(t * 10^(0*4/63)), ... , sin(t * 10^(63*4/63)), cos(t * 10^(0*4/63)), ... , cos(t * 10^(63*4/63))]
    Parameters:
    diffusion_steps (torch.long tensor, shape=(batchsize, 1)):
                                diffusion steps for batch data
    diffusion_step_embed_dim_in (int, default=128):
                                dimensionality of the embedding space for discrete diffusion steps

    Returns:
    the embedding vectors (torch.tensor, shape=(batchsize, diffusion_step_embed_dim_in)):
    """

    assert diffusion_step_embed_dim_in % 2 == 0

    half_dim = diffusion_step_embed_dim_in // 2
    _embed = np.log(10000) / (half_dim - 1)
    _embed = torch.exp(torch.arange(half_dim) * -_embed).to(device)
    _embed = diffusion_steps * _embed
    diffusion_step_embed = torch.cat((torch.sin(_embed), torch.cos(_embed)), 1)

    return diffusion_step_embed

calc_diffusion_hyperparams

calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T)

Compute diffusion process hyperparameters

Parameters: T (int): number of diffusion steps beta_0 and beta_T (float): beta schedule start/end value, where any beta_t in the middle is linearly interpolated

a dictionary of diffusion hyperparameters including: T (int), Beta/Alpha/Alpha_bar/Sigma (torch.tensor on cpu, shape=(T, )) These cpu tensors are changed to cuda tensors on each individual gpu

Source code in icu_benchmarks/imputation/sssds4.py
def calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T):
    """
    Compute diffusion process hyperparameters

    Parameters:
    T (int):                    number of diffusion steps
    beta_0 and beta_T (float):  beta schedule start/end value,
                                where any beta_t in the middle is linearly interpolated

    Returns:
    a dictionary of diffusion hyperparameters including:
        T (int), Beta/Alpha/Alpha_bar/Sigma (torch.tensor on cpu, shape=(T, ))
        These cpu tensors are changed to cuda tensors on each individual gpu
    """

    Beta = torch.linspace(beta_0, beta_T, diffusion_time_steps)  # Linear schedule
    Alpha = 1 - Beta
    Alpha_bar = Alpha + 0
    Beta_tilde = Beta + 0
    for t in range(1, diffusion_time_steps):
        Alpha_bar[t] *= Alpha_bar[t - 1]  # \bar{\alpha}_t = \prod_{s=1}^t \alpha_s
        Beta_tilde[t] *= (1 - Alpha_bar[t - 1]) / (1 - Alpha_bar[t])  # \tilde{\beta}_t = \beta_t * (1-\bar{\alpha}_{t-1})
        # / (1-\bar{\alpha}_t)
    Sigma = torch.sqrt(Beta_tilde)  # \sigma_t^2  = \tilde{\beta}_t

    _dh = {}
    _dh["T"], _dh["Beta"], _dh["Alpha"], _dh["Alpha_bar"], _dh["Sigma"] = (
        diffusion_time_steps,
        Beta,
        Alpha,
        Alpha_bar,
        Sigma,
    )
    diffusion_hyperparams = _dh
    return diffusion_hyperparams

icu_benchmarks.imputation.sssdsa

Classes:

Name Description
SSSDSA
DownPool
UpPool
FFBlock
ResidualBlock

Functions:

Name Description
std_normal

Generate the standard Gaussian variable of a certain size

calc_diffusion_step_embedding

Embed a diffusion step $t$ into a higher dimensional space

largets_component

returns the prime number that divides number the most times.

next_power

returns the next power of 2.

calc_diffusion_hyperparams

Compute diffusion process hyperparameters

SSSDSA

SSSDSA(d_model, n_layers, pool, expand, ff, glu, unet, dropout, in_channels, out_channels, diffusion_step_embed_dim_in, diffusion_step_embed_dim_mid, diffusion_step_embed_dim_out, label_embed_dim, label_embed_classes, bidirectional, s4_lmax, s4_d_state, s4_dropout, s4_bidirectional, diffusion_time_steps, beta_0, beta_T, input_size, *args, **kwargs)

Bases: ImputationWrapper

"SaShiMi model backbone. Adapted from https://github.com/AI4HealthUOL/SSSD/blob/main/src/imputers/SSSDSAImputer.py

SaShiMi model backbone.

Parameters:

Name Type Description Default
d_model

dimension of the model. We generally use 64 for all our experiments.

required
n_layers

number of (Residual (S4) --> Residual (FF)) blocks at each pooling level. We use 8 layers for our experiments, although we found that increasing layers even further generally improves performance at the expense of training / inference speed.

required
pool

pooling factor at each level. Pooling shrinks the sequence length at lower levels. We experimented with a pooling factor of 4 with 1 to 4 tiers of pooling and found 2 tiers to be best. It's possible that a different combination of pooling factors and number of tiers may perform better.

required
expand

expansion factor when pooling. Features are expanded (i.e. the model becomes wider) at lower levels of the architecture.We generally found 2 to perform best (among 2, 4).

required
ff

expansion factor for the FF inverted bottleneck. We generally found 2 to perform best (among 2, 4).

required
bidirectional

use bidirectional S4 layers. Bidirectional layers are suitable for use with non-causal models such as diffusion models like DiffWave.

required
glu

use gated linear unit in the S4 layers. Adds parameters and generally improves performance.

required
unet

use a unet-like architecture, adding (Residual (S4) --> Residual (FF)) layers before downpooling. All else fixed, this slows down inference (and slightly slows training), but generally improves performance. We use this variant when dropping in SaShiMi into diffusion models, and this should generally be preferred for non-autoregressive models.

required
dropout

dropout rate. Default to 0.0, since we haven't found settings where SaShiMi overfits.

required

Methods:

Name Description
step

input: (batch, d_input)

setup_rnn

Convert the SaShiMi model to a RNN for autoregressive generation.

sampling

Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{ heta}(x_{t-1}|x_t)

Source code in icu_benchmarks/imputation/sssdsa.py
def __init__(
    self,
    d_model,
    n_layers,
    pool,
    expand,
    ff,
    glu,
    unet,
    dropout,
    in_channels,
    out_channels,
    diffusion_step_embed_dim_in,
    diffusion_step_embed_dim_mid,
    diffusion_step_embed_dim_out,
    label_embed_dim,
    label_embed_classes,
    bidirectional,
    s4_lmax,
    s4_d_state,
    s4_dropout,
    s4_bidirectional,
    diffusion_time_steps,
    beta_0,
    beta_T,
    input_size,
    *args,
    **kwargs,
):
    """
    SaShiMi model backbone.

    Args:
        d_model: dimension of the model. We generally use 64 for all our experiments.
        n_layers: number of (Residual (S4) --> Residual (FF)) blocks at each pooling level.
            We use 8 layers for our experiments, although we found that increasing layers even further generally
            improves performance at the expense of training / inference speed.
        pool: pooling factor at each level. Pooling shrinks the sequence length at lower levels.
            We experimented with a pooling factor of 4 with 1 to 4 tiers of pooling and found 2 tiers to be best.
            It's possible that a different combination of pooling factors and number of tiers may perform better.
        expand: expansion factor when pooling. Features are expanded (i.e. the model becomes wider) at lower levels of the
            architecture.We generally found 2 to perform best (among 2, 4).
        ff: expansion factor for the FF inverted bottleneck. We generally found 2 to perform best (among 2, 4).
        bidirectional: use bidirectional S4 layers. Bidirectional layers are suitable for use with non-causal models
            such as diffusion models like DiffWave.
        glu: use gated linear unit in the S4 layers. Adds parameters and generally improves performance.
        unet: use a unet-like architecture, adding (Residual (S4) --> Residual (FF)) layers before downpooling.
            All else fixed, this slows down inference (and slightly slows training), but generally improves performance.
            We use this variant when dropping in SaShiMi into diffusion models, and this should generally be preferred
            for non-autoregressive models.
        dropout: dropout rate. Default to 0.0, since we haven't found settings where SaShiMi overfits.
    """
    super().__init__(
        d_model=d_model,
        n_layers=n_layers,
        pool=pool,
        expand=expand,
        ff=ff,
        glu=glu,
        unet=unet,
        dropout=dropout,
        in_channels=in_channels,
        out_channels=out_channels,
        diffusion_step_embed_dim_in=diffusion_step_embed_dim_in,
        diffusion_step_embed_dim_mid=diffusion_step_embed_dim_mid,
        diffusion_step_embed_dim_out=diffusion_step_embed_dim_out,
        label_embed_dim=label_embed_dim,
        label_embed_classes=label_embed_classes,
        bidirectional=bidirectional,
        s4_lmax=s4_lmax,
        s4_d_state=s4_d_state,
        s4_dropout=s4_dropout,
        s4_bidirectional=s4_bidirectional,
        diffusion_time_steps=diffusion_time_steps,
        beta_0=beta_0,
        beta_T=beta_T,
        input_size=input_size,
        *args,
        **kwargs,
    )
    self.diffusion_parameters = calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T)
    self.d_model = H = d_model
    self.unet = unet

    def s4_block(dim, stride):
        layer = S4(
            d_model=dim,
            l_max=s4_lmax,
            d_state=s4_d_state,
            bidirectional=s4_bidirectional,
            postact="glu" if glu else None,
            dropout=dropout,
            transposed=True,
            # hurwitz=True, # use the Hurwitz parameterization for stability
            # tie_state=True, # tie SSM parameters across d_state in the S4 layer
            trainable={
                "dt": True,
                "A": True,
                "P": True,
                "B": True,
            },  # train all internal S4 parameters
        )

        return ResidualBlock(
            d_model=dim,
            layer=layer,
            dropout=dropout,
            diffusion_step_embed_dim_out=diffusion_step_embed_dim_out,
            in_channels=in_channels,
            label_embed_dim=label_embed_dim,
            stride=stride,
        )

    def ff_block(dim, stride):
        layer = FFBlock(
            d_model=dim,
            expand=ff,
            dropout=dropout,
        )
        return ResidualBlock(
            d_model=dim,
            layer=layer,
            dropout=dropout,
            diffusion_step_embed_dim_out=diffusion_step_embed_dim_out,
            in_channels=in_channels,
            label_embed_dim=label_embed_dim,
            stride=stride,
        )

    # Down blocks
    d_layers, H = self.init_down_blocks(pool, unet, n_layers, ff, H, expand, s4_block, ff_block)

    # Center block
    c_layers = self.init_center_blocks(pool, n_layers, ff, H, s4_block, ff_block)

    # Up blocks
    u_layers, H = self.init_up_blocks(pool, n_layers, ff, H, expand, bidirectional, s4_block, ff_block)

    self.d_layers = nn.ModuleList(d_layers)
    self.c_layers = nn.ModuleList(c_layers)
    self.u_layers = nn.ModuleList(u_layers)
    self.norm = nn.LayerNorm(H)

    self.init_conv = nn.Sequential(nn.Conv1d(in_channels, d_model, kernel_size=1), nn.ReLU())
    self.final_conv = nn.Sequential(
        nn.Conv1d(d_model, d_model, kernel_size=1),
        nn.ReLU(),
        nn.Conv1d(d_model, out_channels, kernel_size=1),
    )
    self.fc_t1 = nn.Linear(diffusion_step_embed_dim_in, diffusion_step_embed_dim_mid)
    self.fc_t2 = nn.Linear(diffusion_step_embed_dim_mid, diffusion_step_embed_dim_out)
    self.cond_embedding = nn.Embedding(label_embed_classes, label_embed_dim)
    self.diffusion_step_embed_dim_in = diffusion_step_embed_dim_in

    assert H == d_model

step

step(x, state, **kwargs)

input: (batch, d_input) output: (batch, d_output)

Source code in icu_benchmarks/imputation/sssdsa.py
def step(self, x, state, **kwargs):
    """
    input: (batch, d_input)
    output: (batch, d_output)
    """
    # States will be popped in reverse order for convenience
    state = state[::-1]

    # Down blocks
    outputs = []  # Store all layers for SaShiMi
    next_state = []
    for layer in self.d_layers:
        outputs.append(x)
        x, _next_state = layer.step(x, state=state.pop(), **kwargs)
        next_state.append(_next_state)
        if x is None:
            break

    # Center block
    if x is None:
        # Skip computations since we've downsized
        skipped = len(self.d_layers) - len(outputs)
        for _ in range(skipped + len(self.c_layers)):
            next_state.append(state.pop())
        if self.unet:
            for i in range(skipped):
                next_state.append(state.pop())
            u_layers = list(self.u_layers)[skipped // 3 :]
        else:
            for i in range(skipped):
                for _ in range(len(self.u_layers[i])):
                    next_state.append(state.pop())
            u_layers = list(self.u_layers)[skipped:]
    else:
        outputs.append(x)
        for layer in self.c_layers:
            x, _next_state = layer.step(x, state=state.pop(), **kwargs)
            next_state.append(_next_state)
        x = x + outputs.pop()
        u_layers = self.u_layers

    x, next_state = self.up_blocks_loop(x, u_layers, next_state, state, outputs, **kwargs)

    # feature projection
    x = self.norm(x)
    return x, next_state

setup_rnn

setup_rnn(mode='dense')

Convert the SaShiMi model to a RNN for autoregressive generation.

Parameters:

Name Type Description Default
mode

S4 recurrence mode. Using diagonal can speed up generation by 10-20%. linear should be faster theoretically but is slow in practice since it dispatches more operations (could benefit from fused operations). Note that diagonal could potentially be unstable if the diagonalization is numerically unstable (although we haven't encountered this case in practice), while dense should always be stable.

'dense'
Source code in icu_benchmarks/imputation/sssdsa.py
def setup_rnn(self, mode="dense"):
    """
    Convert the SaShiMi model to a RNN for autoregressive generation.

    Args:
        mode: S4 recurrence mode. Using `diagonal` can speed up generation by 10-20%.
            `linear` should be faster theoretically but is slow in practice since it
            dispatches more operations (could benefit from fused operations).
            Note that `diagonal` could potentially be unstable if the diagonalization is numerically unstable
            (although we haven't encountered this case in practice), while `dense` should always be stable.
    """
    assert mode in ["dense", "diagonal", "linear"]
    for module in self.modules():
        if hasattr(module, "setup_step"):
            module.setup_step(mode)

sampling

sampling(cond, mask)

Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{ heta}(x_{t-1}|x_t)

Parameters: net (torch network): the wavenet model size (tuple): size of tensor to be generated, usually is (number of audios to generate, channels=1, length of audio) diffusion_hyperparams (dict): dictionary of diffusion hyperparameters returned by calc_diffusion_hyperparams note, the tensors need to be cuda tensors

Returns: the generated audio(s) in torch.tensor, shape=size

Source code in icu_benchmarks/imputation/sssdsa.py
def sampling(self, cond, mask):
    """
    Perform the complete sampling step according to p(x_0|x_T) = prod_{t=1}^T p_{\theta}(x_{t-1}|x_t)

    Parameters:
    net (torch network):            the wavenet model
    size (tuple):                   size of tensor to be generated,
                                    usually is (number of audios to generate, channels=1, length of audio)
    diffusion_hyperparams (dict):   dictionary of diffusion hyperparameters returned by calc_diffusion_hyperparams
                                    note, the tensors need to be cuda tensors

    Returns:
    the generated audio(s) in torch.tensor, shape=size
    """

    Alpha, Alpha_bar, Sigma = (
        self.diffusion_parameters["Alpha"],
        self.diffusion_parameters["Alpha_bar"],
        self.diffusion_parameters["Sigma"],
    )

    T = self.hparams.diffusion_time_steps
    assert len(Alpha) == T
    assert len(Alpha_bar) == T
    assert len(Sigma) == T

    B, _, _ = cond.shape
    x = std_normal(cond.shape, self.device)

    for t in range(T - 1, -1, -1):
        x = x * (1 - mask).float() + cond * mask.float()
        diffusion_steps = (t * torch.ones((B, 1))).to(self.device)  # use the corresponding reverse step
        epsilon_theta = self(
            (
                x,
                cond,
                mask,
                diffusion_steps,
            )
        )  # predict \epsilon according to \epsilon_\theta
        # update x_{t-1} to \mu_\theta(x_t)
        x = (x - (1 - Alpha[t]) / torch.sqrt(1 - Alpha_bar[t]) * epsilon_theta) / torch.sqrt(Alpha[t])
        if t > 0:
            x = x + Sigma[t] * std_normal(cond.shape, self.device)  # add the variance term to x_{t-1}

    return x

DownPool

DownPool(d_input, expand, pool)

Bases: Module

Methods:

Name Description
step

x: (..., H)

Source code in icu_benchmarks/imputation/sssdsa.py
def __init__(self, d_input, expand, pool):
    super().__init__()
    self.d_input = d_input
    self.d_output = d_input * expand
    self.pool = pool

    self.linear = LinearActivation(
        d_input * pool,
        self.d_output,
        transposed=True,
        weight_norm=True,
    )

step

step(x, state, **kwargs)

x: (..., H)

Source code in icu_benchmarks/imputation/sssdsa.py
def step(self, x, state, **kwargs):
    """
    x: (..., H)
    """

    if x is None:
        return None, state
    state.append(x)
    if len(state) == self.pool:
        x = rearrange(torch.stack(state, dim=-1), "... h s -> ... (h s)")
        x = x.unsqueeze(-1)
        x = self.linear(x)
        x = x.squeeze(-1)
        return x, []
    else:
        return None, state

UpPool

UpPool(d_input, expand, pool, causal=True)

Bases: Module

Methods:

Name Description
step

x: (..., H)

Source code in icu_benchmarks/imputation/sssdsa.py
def __init__(self, d_input, expand, pool, causal=True):
    super().__init__()
    self.d_output = d_input // expand
    self.pool = pool
    self.causal = causal

    self.linear = LinearActivation(
        d_input,
        self.d_output * pool,
        transposed=True,
        weight_norm=True,
    )

step

step(x, state, **kwargs)

x: (..., H)

Source code in icu_benchmarks/imputation/sssdsa.py
def step(self, x, state, **kwargs):
    """
    x: (..., H)
    """
    assert len(state) > 0
    y, state = state[0], state[1:]
    if len(state) == 0:
        assert x is not None
        x = x.unsqueeze(-1)
        x = self.linear(x)
        x = x.squeeze(-1)
        x = rearrange(x, "... (h s) -> ... h s", s=self.pool)
        state = list(torch.unbind(x, dim=-1))
    else:
        assert x is None
    return y, state

FFBlock

FFBlock(d_model, expand=2, dropout=0.0)

Bases: Module

Feed-forward block.

Parameters:

Name Type Description Default
d_model

dimension of input

required
expand

expansion factor for inverted bottleneck

2
dropout

dropout rate

0.0
Source code in icu_benchmarks/imputation/sssdsa.py
def __init__(self, d_model, expand=2, dropout=0.0):
    """
    Feed-forward block.

    Args:
        d_model: dimension of input
        expand: expansion factor for inverted bottleneck
        dropout: dropout rate
    """
    super().__init__()

    input_linear = LinearActivation(
        d_model,
        d_model * expand,
        transposed=True,
        activation="gelu",
        activate=True,
    )
    dropout = nn.Dropout2d(dropout) if dropout > 0.0 else nn.Identity()
    output_linear = LinearActivation(
        d_model * expand,
        d_model,
        transposed=True,
        activation=None,
        activate=False,
    )

    self.ff = nn.Sequential(
        input_linear,
        dropout,
        output_linear,
    )

ResidualBlock

ResidualBlock(d_model, layer, dropout, diffusion_step_embed_dim_out, in_channels, label_embed_dim, stride)

Bases: Module

Residual S4 block.

Parameters:

Name Type Description Default
d_model

dimension of the model

required
bidirectional

use bidirectional S4 layer

required
glu

use gated linear unit in the S4 layer

required
dropout

dropout rate

required

Methods:

Name Description
forward

Input x is shape (B, d_input, L)

Source code in icu_benchmarks/imputation/sssdsa.py
def __init__(
    self,
    d_model,
    layer,
    dropout,
    diffusion_step_embed_dim_out,
    in_channels,
    label_embed_dim,
    stride,
):
    """
    Residual S4 block.

    Args:
        d_model: dimension of the model
        bidirectional: use bidirectional S4 layer
        glu: use gated linear unit in the S4 layer
        dropout: dropout rate
    """
    super().__init__()

    self.layer = layer
    self.norm = nn.LayerNorm(d_model)
    self.dropout = nn.Dropout2d(dropout) if dropout > 0.0 else nn.Identity()

    self.fc_t = nn.Linear(diffusion_step_embed_dim_out, d_model)
    self.cond_conv = Conv(2 * in_channels, d_model, kernel_size=stride, stride=stride)
    self.fc_label = nn.Linear(label_embed_dim, d_model) if label_embed_dim is not None else None

forward

forward(input_data)

Input x is shape (B, d_input, L)

Source code in icu_benchmarks/imputation/sssdsa.py
def forward(self, input_data):
    """
    Input x is shape (B, d_input, L)
    """
    x, cond, diffusion_step_embed = input_data

    # add in diffusion step embedding
    part_t = self.fc_t(diffusion_step_embed).unsqueeze(2)
    z = x + part_t

    # Prenorm
    z = self.norm(z.transpose(-1, -2)).transpose(-1, -2)

    z, _ = self.layer(z)

    cond = self.cond_conv(cond)
    # cond = self.fc_label(cond)

    z = z + cond

    # Dropout on the output of the layer
    z = self.dropout(z)

    # Residual connection
    x = z + x

    return x

std_normal

std_normal(size, device)

Generate the standard Gaussian variable of a certain size

Source code in icu_benchmarks/imputation/sssdsa.py
def std_normal(size, device):
    """
    Generate the standard Gaussian variable of a certain size
    """

    return torch.normal(0, 1, size=size).to(device)

calc_diffusion_step_embedding

calc_diffusion_step_embedding(diffusion_steps, diffusion_step_embed_dim_in, device)

Embed a diffusion step $t$ into a higher dimensional space E.g. the embedding vector in the 128-dimensional space is [sin(t * 10^(04/63)), ... , sin(t * 10^(634/63)), cos(t * 10^(04/63)), ... , cos(t * 10^(634/63))] Parameters: diffusion_steps (torch.long tensor, shape=(batchsize, 1)): diffusion steps for batch data diffusion_step_embed_dim_in (int, default=128): dimensionality of the embedding space for discrete diffusion steps

Returns: the embedding vectors (torch.tensor, shape=(batchsize, diffusion_step_embed_dim_in)):

Source code in icu_benchmarks/imputation/sssdsa.py
def calc_diffusion_step_embedding(diffusion_steps, diffusion_step_embed_dim_in, device):
    """
    Embed a diffusion step $t$ into a higher dimensional space
    E.g. the embedding vector in the 128-dimensional space is
    [sin(t * 10^(0*4/63)), ... , sin(t * 10^(63*4/63)), cos(t * 10^(0*4/63)), ... , cos(t * 10^(63*4/63))]
    Parameters:
    diffusion_steps (torch.long tensor, shape=(batchsize, 1)):
                                diffusion steps for batch data
    diffusion_step_embed_dim_in (int, default=128):
                                dimensionality of the embedding space for discrete diffusion steps

    Returns:
    the embedding vectors (torch.tensor, shape=(batchsize, diffusion_step_embed_dim_in)):
    """

    assert diffusion_step_embed_dim_in % 2 == 0

    half_dim = diffusion_step_embed_dim_in // 2
    _embed = np.log(10000) / (half_dim - 1)
    _embed = torch.exp(torch.arange(half_dim) * -_embed).to(device)
    _embed = diffusion_steps * _embed
    diffusion_step_embed = torch.cat((torch.sin(_embed), torch.cos(_embed)), 1)

    return diffusion_step_embed

largets_component

largets_component(number)

returns the prime number that divides number the most times.

Source code in icu_benchmarks/imputation/sssdsa.py
def largets_component(number):
    """
    returns the prime number that divides number the most times.
    """
    for i in range(2, int(number**0.5) + 1):
        if number % i == 0:
            return i
    return number

next_power

next_power(number)

returns the next power of 2.

Source code in icu_benchmarks/imputation/sssdsa.py
def next_power(number):
    """
    returns the next power of 2.
    """
    return 1 << (number - 1).bit_length()

calc_diffusion_hyperparams

calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T)

Compute diffusion process hyperparameters

Parameters:

Name Type Description Default
T int

number of diffusion steps.

required
beta_0 beta_T (float)

beta schedule start/end value, where any beta_t in the middle is linearly interpolated

required

a dictionary of diffusion hyperparameters including: T (int), Beta/Alpha/Alpha_bar/Sigma (torch.tensor on cpu, shape=(T, )) These cpu tensors are changed to cuda tensors on each individual gpu

Source code in icu_benchmarks/imputation/sssdsa.py
def calc_diffusion_hyperparams(diffusion_time_steps, beta_0, beta_T):
    """
    Compute diffusion process hyperparameters

    Params:
        T (int): number of diffusion steps.
        beta_0 beta_T (float) :beta schedule start/end value, where any beta_t in the middle is linearly interpolated

    Returns:
    a dictionary of diffusion hyperparameters including:
        T (int), Beta/Alpha/Alpha_bar/Sigma (torch.tensor on cpu, shape=(T, ))
        These cpu tensors are changed to cuda tensors on each individual gpu
    """

    Beta = torch.linspace(beta_0, beta_T, diffusion_time_steps)  # Linear schedule
    Alpha = 1 - Beta
    Alpha_bar = Alpha + 0
    Beta_tilde = Beta + 0
    for t in range(1, diffusion_time_steps):
        Alpha_bar[t] *= Alpha_bar[t - 1]  # \bar{\alpha}_t = \prod_{s=1}^t \alpha_s
        Beta_tilde[t] *= (1 - Alpha_bar[t - 1]) / (1 - Alpha_bar[t])  # \tilde{\beta}_t = \beta_t * (1-\bar{\alpha}_{t-1})
        # / (1-\bar{\alpha}_t)
    Sigma = torch.sqrt(Beta_tilde)  # \sigma_t^2  = \tilde{\beta}_t

    _dh = {}
    _dh["T"], _dh["Beta"], _dh["Alpha"], _dh["Alpha_bar"], _dh["Sigma"] = (
        diffusion_time_steps,
        Beta,
        Alpha,
        Alpha_bar,
        Sigma,
    )
    diffusion_hyperparams = _dh
    return diffusion_hyperparams