SPD linear algebra

yetanotherspdnet.functions.spd_linalg — matrix functions computed through the eigendecomposition \(X = U \operatorname{diag}(\lambda) U^\top\), \(f(X) = U \operatorname{diag}(f(\lambda)) U^\top\), plus congruences, whitening and vectorizations. Every operation exists twice: a function differentiated by autograd, and a torch.autograd.Function whose backward is written by hand (Daleckii–Krein formula), which stays exact and finite when eigenvalues are close or repeated. Layers pick one with use_autograd (manual by default); see Numerical and optimization techniques.

Functions returning a matrix built from an eigendecomposition (sqrtm_SPD, logm_SPD, …) return the tuple (result, eigvals, eigvecs): index [0] for the matrix. The Function classes return the matrix only.

autograd path	manual backward
symmetrize()	–	Symmetrize a tensor along the last two dimensions.
vec_batch()	VecBatch	Vectorize a batch of tensors along last two dimensions
unvec_batch()	–	Unvectorize a batch of tensors along last dimension
vech_batch()	VechBatch	Vectorize the lower triangular part of a batch of square matrices
unvech_batch()	–	Unvectorize a batch of tensors along last dimension, assuming that matrices are symmetric
eigh_operation()	–	Applies a function on the eigenvalues of a batch of symmetric matrices.
eigh_operation_grad()	–	Computes the backpropagation of the gradient for a function applied on the eigenvalues of a batch of symmetric matrices
solve_sylvester_SPD()	–	Solve Sylvester equations in the context of SPD matrices relying on eigenvalue decomposition.
sqrtm_SPD()	SqrtmSPD	Matrix square root of a batch of SPD matrices.
inv_sqrtm_SPD()	InvSqrtmSPD	Inverse matrix square root of a batch of SPD matrices.
powm_SPD()	PowmSPD	Matrix power of a batch of SPD matrices.
logm_SPD()	LogmSPD	Matrix logarithm of a batch of SPD matrices.
expm_symmetric()	ExpmSymmetric	Matrix exponential of a batch of symmetric matrices.
scaled_softplus_symmetric()	ScaledSoftPlusSymmetric	Scaled matrix SoftPlus of a batch of symmetric matrices.
inv_scaled_softplus_SPD()	InvScaledSoftPlusSPD	Inverse scaled SoftPlus of a batch of SPD matrices.
eigh_relu()	EighReLu	ReLu activation function on the eigenvalues of SPD matrices.
eigh_relu_bias()	EighReLuBias	Eigenvalue rectification with a learnable shift of the eigenvalues.
congruence_SPD()	CongruenceSPD	Congruence of a batch of SPD matrices with an SPD matrix.
whitening()	Whitening	Whitening of a batch of SPD matrices with an SPD matrix.
congruence_rectangular()	CongruenceRectangular	Forward pass of the congruence of a batch of SPD matrices with a (full-rank) rectangular matrix.

Details

Core SPD matrix linear algebra: eigendecomposition, matrix functions, congruence, and vectorization.

symmetrize(data)

Symmetrize a tensor along the last two dimensions.

\[\operatorname{sym}(A) = \frac{1}{2}\big(A + A^\top\big)\]

Parameters:

data (torch.Tensor of shape (..., n_features, n_features)) – Batch of square matrices

Returns:

sym_data (torch.Tensor of shape (..., n_features, n_features)) – Batch of symmetrized matrices

Return type:

Tensor

vec_batch(data)

Vectorize a batch of tensors along last two dimensions

Parameters:

data (torch.Tensor of shape (..., n_rows, n_columns)) – Batch of matrices

Returns:

data_vec (torch.Tensor of shape (..., n_rows*n_columns)) – Batch of vectorized matrices

Return type:

Tensor

unvec_batch(data_vec, n_rows)

Unvectorize a batch of tensors along last dimension

Parameters:

• data_vec (torch.Tensor of shape (..., n_rows*n_columns)) – Batch of vectorized matrices

• n_rows (int) – Number of rows of the matrices

Returns:

data (torch.Tensor of shape (..., n_rows, n_columns)) – Batch of matrices

Return type:

Tensor

class VecBatch(*args, **kwargs)

Vectorize a batch of matrices along last two dimensions. Matrices are assumed to be symmetric (for backward)

static forward(ctx, data)

Forward pass of the vectorization of a batch of matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

vec_data (torch.Tensor of shape (..., n_features ** 2))

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the vectorization of a batch of symmetric matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features ** 2)) – Gradient of the loss with respect to vectorized input batch of matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of matrices

Return type:

Tensor

vech_batch(data)

Vectorize the lower triangular part of a batch of square matrices

Parameters:

data (torch.Tensor of shape (..., n_features, n_features)) – Batch of matrices

Returns:

data_vech (torch.Tensor of shape (..., n_features*(n_features+1)//2)) – Batch of vectorized matrices

Return type:

Tensor

unvech_batch(data_vech, n_features)

Unvectorize a batch of tensors along last dimension, assuming that matrices are symmetric

Parameters:

• X_vech (torch.Tensor of shape (..., n_features*(n_features+1)//2)) – Batch of vectorized matrices

• n_features (int) – number of features

Returns:

X (torch.Tensor of shape (..., n_features, n_features)) – Batch of symmetric matrices

Return type:

Tensor

class VechBatch(*args, **kwargs)

Half vectorize a batch of symmetric matrices along last two dimensions

static forward(ctx, data)

Forward pass of the half vectorization of a batch of matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

vech_data (torch.Tensor of shape (..., n_features*(n_features+1) // 2))

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the vectorization of a batch of matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features*(n_features+1) // 2)) – Gradient of the loss with respect to half vectorized input batch of symmetric matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of symmetric matrices

Return type:

Tensor

eigh_operation(eigvals, eigvecs, operation)

Applies a function on the eigenvalues of a batch of symmetric matrices.

\[f(A) = V \operatorname{diag}\big(f(\lambda_1), \dots, f(\lambda_n)\big) V^\top\]

given the eigendecomposition \(A = V \operatorname{diag}(\lambda) V^\top\). This is the core primitive behind every matrix function in this module (sqrtm_SPD(), inv_sqrtm_SPD(), powm_SPD(), logm_SPD(), expm_symmetric(), …): each just picks a different scalar operation applied eigenvalue-wise.

Parameters:

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of the corresponding batch of symmetric matrices

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of the corresponding batch of symmetric matrices

• operation (Callable) – Function to apply on eigenvalues

Returns:

result (torch.Tensor of shape (..., n_features, n_features)) – Resulting symmetric matrices with operation applied to eigenvalues

Return type:

Tensor

eigh_operation_grad(grad_output, eigvals, eigvecs, operation, operation_deriv)

Computes the backpropagation of the gradient for a function applied on the eigenvalues of a batch of symmetric matrices

Parameters:

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the output of the operation on eigenvalues

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of the corresponding batch of symmetric matrices

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of the corresponding batch of symmetric matrices

• operation (Callable) – Function to apply on eigenvalues

• operation_deriv (Callable) – Derivative of the function to apply on eigenvalues

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of symmetric matrices

Return type:

Tensor

solve_sylvester_SPD(eigvals, eigvecs, mat)

Solve Sylvester equations in the context of SPD matrices relying on eigenvalue decomposition.

Given \(A = V \operatorname{diag}(\lambda) V^\top\) (via eigvals, eigvecs), solves \(AX + XA = \text{mat}\) for \(X\) in closed form:

\[X = V\left[\frac{1}{\lambda_i + \lambda_j} \,(V^\top \,\text{mat}\, V)_{ij}\right]_{ij} V^\top\]

Parameters:

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of a batch of SPD matrices

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of a batch of SPD matrices

• mat (torch.Tensor of shape (..., n_features, n_features)) – Batch of matrices on the right side of Sylvester equations. If matrices are symmetric then the solutions will be symmetric. If they are skew-symmetric, then the results will be skew-symmetric.

Returns:

result (torch.Tensor of shape (..., n_features, n_features)) – Symmetric matrices solutions to Sylvester equations

Return type:

Tensor

sqrtm_SPD(data)

Matrix square root of a batch of SPD matrices.

\[P^{1/2} = V \operatorname{diag}(\sqrt{\lambda}) V^\top\]

via eigh_operation() with operation=torch.sqrt.

Parameters:

data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

• sqrtm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix square roots of the input batch of SPD matrices

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class SqrtmSPD(*args, **kwargs)

Matrix square root of a batch of SPD matrices (relies on eigenvalue decomposition)

static forward(ctx, data)

Forward pass of the matrix square root of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

sqrtm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix square roots of the input batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the matrix square root of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to matrix square roots of the input batch of SPD matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of SPD matrices

Return type:

Tensor

inv_sqrtm_SPD(data)

Inverse matrix square root of a batch of SPD matrices.

\[P^{-1/2} = V \operatorname{diag}(\lambda^{-1/2}) V^\top\]

via eigh_operation() with operation=inv_sqrt.

Parameters:

data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

• inv_sqrtm_data (torch.Tensor of shape (..., n_features, n_features)) – Inverse matrix square roots of the input batch of SPD matrices

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class InvSqrtmSPD(*args, **kwargs)

Matrix inverse square root of a batch of SPD matrices (relies on eigenvalue decomposition)

static forward(ctx, data)

Forward pass of the matrix inverse square root of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

inv_sqrtm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix square roots of the input batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the matrix inverse square root of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to matrix square roots of the input batch of SPD matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of SPD matrices

Return type:

Tensor

powm_SPD(data, exponent)

Matrix power of a batch of SPD matrices.

\[P^{p} = V \operatorname{diag}(\lambda^{p}) V^\top\]

via eigh_operation() with operation=lambda x: x**exponent.

Parameters:

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• exponent (torch.float) – Power exponent

Returns:

• powm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix powers of the input batch of SPD matrices

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class PowmSPD(*args, **kwargs)

Matrix power of a batch of SPD matrices (relies on eigenvalue decomposition)

static forward(ctx, data, exponent)

Forward pass of the matrix power of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• exponent (torch.float) – Power exponent

Returns:

powm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix powers of the input batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the matrix power of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to matrix powers of the input batch of SPD matrices

Returns:

• grad_input_data (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of SPD matrices

• grad_input_exponent (torch.float) – Gradient of the loss with respect to the power exponent

Return type:

tuple[Tensor, Tensor]

logm_SPD(data)

Matrix logarithm of a batch of SPD matrices.

\[\log(P) = V \operatorname{diag}(\log\lambda) V^\top\]

via eigh_operation() with operation=torch.log. Maps the SPD manifold to the vector space of symmetric matrices — the basis of the Log-Euclidean geometry (see log_euclidean).

Parameters:

data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

• logm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix logarithms of the input batch of SPD matrices

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class LogmSPD(*args, **kwargs)

Matrix logarithm of a batch of SPD matrices (relies on eigenvalue decomposition)

static forward(ctx, data)

Forward pass of the matrix logarithm of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

logm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix logarithms of the input batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the matrix logarithm of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to matrix logarithms of the input batch of SPD matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of SPD matrices

Return type:

Tensor

expm_symmetric(data)

Matrix exponential of a batch of symmetric matrices.

\[\exp(S) = V \operatorname{diag}(\exp\lambda) V^\top\]

via eigh_operation() with operation=torch.exp. The result is always SPD (eigenvalues \(\exp\lambda > 0\)); inverse of logm_SPD().

Parameters:

data (torch.Tensor of shape (..., n_features, n_features)) – Batch of symmetric matrices

Returns:

• expm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix exponentials of the input batch of symmetric matrices

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class ExpmSymmetric(*args, **kwargs)

Matrix exponential of a batch of symmetric matrices (relies on eigenvalue decomposition)

static forward(ctx, data)

Forward pass of the matrix exponential of a batch of symmetric matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of symmetric matrices

Returns:

expm_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix exponentials of the input batch of symmetric matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the matrix exponential of a batch of symmetric matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to matrix exponentials of the input batch of symmetric matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of symmetric matrices

Return type:

Tensor

scaled_softplus_symmetric(data)

Scaled matrix SoftPlus of a batch of symmetric matrices.

\[f(S) = V \operatorname{diag}\big(\log_2(1 + 2^{\lambda})\big) V^\top\]

via eigh_operation() with operation= scaled_softplus(). Maps any symmetric matrix to an SPD matrix (eigenvalues strictly positive) — used to parametrize BiMap weights or BatchNorm scale so they stay on the SPD manifold under unconstrained optimization.

Parameters:

data (torch.Tensor of shape (..., n_features, n_features)) – Batch of symmetric matrices

Returns:

• softplus_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix SoftPlus of the input batch of symmetric matrices

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class ScaledSoftPlusSymmetric(*args, **kwargs)

Scaled matrix SoftPlus of a batch of symmetric matrices. It is scaled so that: f(0) = 1, f(x) -> 0 as x -> -inf and f’(x) -> 1 as x -> +inf

static forward(ctx, data)

Forward pass of the scaled matrix SoftPlus of a batch of symmetric matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of symmetric matrices

Returns:

softplus_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix SoftPlus of the input batch of symmetric matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the scaled matrix SoftPlus of a batch of symmetric matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to matrix SoftPlus of the input batch of symmetric matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of symmetric matrices

Return type:

Tensor

inv_scaled_softplus_SPD(data)

Inverse scaled SoftPlus of a batch of SPD matrices.

\[f^{-1}(P) = V \operatorname{diag}\big(\log_2(2^{\lambda} - 1)\big) V^\top\]

via eigh_operation(). Inverse of scaled_softplus_symmetric().

Parameters:

data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

• inv_softplus_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix logarithms of the input batch of SPD matrices

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class InvScaledSoftPlusSPD(*args, **kwargs)

Matrix inverse scaled SoftPlus of a batch of SPD matrices

static forward(ctx, data)

Forward pass of the matrix inverse scaled SoftPlus of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Returns:

inv_softplus_data (torch.Tensor of shape (..., n_features, n_features)) – Matrix logarithms of the input batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the matrix inverse scaled SoftPlus of a batch of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to matrix inverse SoftPlus of the input batch of SPD matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of SPD matrices

Return type:

Tensor

eigh_relu(data, eps)

ReLu activation function on the eigenvalues of SPD matrices.

\[\operatorname{ReEig}_\epsilon(P) = V \operatorname{diag}\big( \max(\lambda, \epsilon)\big) V^\top\]

via eigh_operation(). This is the ReEig layer’s core operation (ReEig): clamps small/negative eigenvalues to \(\epsilon > 0\) to keep the result SPD, the manifold analogue of ReLU rectification.

Parameters:

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• eps (float) – Value for the rectification of the eigenvalues

Returns:

• data_transformed (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices with rectified eigenvalues

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class EighReLu(*args, **kwargs)

ReLu activation function on the eigenvalues of SPD matrices

static forward(ctx, data, eps)

Forward pass of the ReLu activation function on the eigenvalues of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to save tensors for the backward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• eps (float) – Value for the rectification of the eigenvalues

Returns:

data_transformed (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices with rectified eigenvalues

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the ReLu activation function on the eigenvalues of SPD matrices

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to save tensors for the backward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the output batch of SPD matrices

Returns:

grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of SPD matrices

Return type:

tuple[Tensor, None]

eigh_relu_bias(data, bias, eps)

Eigenvalue rectification with a learnable shift of the eigenvalues.

\[\operatorname{ReEigBias}_{\epsilon, b}(P) = V \operatorname{diag}\big( \operatorname{clamp}(\lambda_i + b_i,\ \epsilon,\ 1/\epsilon) \big) V^\top\]

with \(\lambda_1 \le \dots \le \lambda_n\) the eigenvalues in ascending order (as returned by torch.linalg.eigh) and \(b\) a bias vector indexed by eigenvalue rank. The upper clamp bounds the condition number of the output by \(\epsilon^{-2}\).

Unlike eigh_relu(), this is not a spectral function when two eigenvalues coincide while their biases differ: the output then depends on the arbitrary eigenbasis of the repeated eigenvalue, and gradients with respect to data are only defined for distinct eigenvalues.

Parameters:

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of symmetric matrices

• bias (torch.Tensor of shape (n_features,)) – Shift added to the (ascending) eigenvalues

• eps (float) – Lower clamping value; the upper one is 1 / eps

Returns:

• data_transformed (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• eigvals (torch.Tensor of shape (..., n_features)) – Eigenvalues of matrices in data

• eigvecs (torch.Tensor of shape (..., n_features, n_features)) – Eigenvectors of matrices in data

Return type:

tuple[Tensor, Tensor, Tensor]

class EighReLuBias(*args, **kwargs)

Eigenvalue rectification with a learnable shift, with a hand-written backward

static forward(ctx, data, bias, eps)

Forward pass of eigh_relu_bias()

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to save tensors for the backward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of symmetric matrices

• bias (torch.Tensor of shape (n_features,)) – Shift added to the (ascending) eigenvalues

• eps (float) – Lower clamping value; the upper one is 1 / eps

Returns:

data_transformed (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of eigh_relu_bias()

The gradient with respect to data is the Daleckii-Krein formula of eigh_operation_grad(). Only the eigenvalues depend on the bias, so its gradient is the diagonal of \(V^ op G V\) masked by the derivative of the clamp, summed over the batch dimensions.

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to save tensors for the backward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the output batch

Returns:

• grad_input (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch

• grad_bias (torch.Tensor of shape (n_features,)) – Gradient of the loss with respect to the bias

Return type:

tuple[Tensor, Tensor, None]

congruence_SPD(data, matrix)

Congruence of a batch of SPD matrices with an SPD matrix.

\[P' = A\, P\, A\]

(here \(A\) is itself SPD, hence symmetric, so \(A^\top = A\) and there is no separate transpose). Congruence by an SPD matrix preserves the SPD manifold.

Parameters:

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• matrix (torch.Tensor of shape (n_features, n_features)) – SPD matrix

Returns:

data_transformed (torch.Tensor of shape (..., n_features, n_features)) – Transformed batch of SPD matrices

Return type:

Tensor

class CongruenceSPD(*args, **kwargs)

Congruence of a batch of SPD matrices with an SPD matrix

static forward(ctx, data, matrix)

Forward pass of the congruence of a batch of SPD matrices with an SPD matrix

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• matrix (torch.Tensor of shape (n_features, n_features)) – SPD matrix

Returns:

data_transformed (torch.Tensor of shape (..., n_features, n_features)) – Transformed batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the congruence of a batch of SPD matrices with an SPD matrix

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the batch of transformed SPD matrices

Returns:

• grad_input_data (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of SPD matrices

• grad_input_bias (torch.Tensor of shape (n_features, n_features)) – Gradient of the loss with respect to the SPD matrix used for congruence

Return type:

tuple[Tensor, Tensor]

whitening(data, matrix)

Whitening of a batch of SPD matrices with an SPD matrix.

\[P' = A^{-1/2}\, P\, A^{-1/2}\]

i.e. congruence_SPD() with \(A^{-1/2}\) (see inv_sqrtm_SPD()) — transforms data so that \(A\) itself maps to the identity, the SPD analogue of standardizing by the covariance.

Parameters:

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• matrix (torch.Tensor of shape (n_features, n_features)) – SPD matrix

Returns:

data_transformed (torch.Tensor of shape (..., n_features, n_features)) – Transformed batch of SPD matrices

Return type:

Tensor

class Whitening(*args, **kwargs)

Whitening of a batch of SPD matrices with an SPD matrix

static forward(ctx, data, matrix)

Forward pass of the whitening of a batch of SPD matrices with an SPD matrix

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_features, n_features)) – Batch of SPD matrices

• matrix (torch.Tensor of shape (n_features, n_features)) – SPD matrix

Returns:

data_transformed (torch.Tensor of shape (..., n_features, n_features)) – Transformed batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the whitening of a batch of SPD matrices with an SPD matrix

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the batch of whitened SPD matrices

Returns:

• grad_input_data (torch.Tensor of shape (..., n_features, n_features)) – Gradient of the loss with respect to the input batch of SPD matrices

• grad_input_matrix (torch.Tensor of shape (n_features, n_features)) – Gradient of the loss with respect to the SPD matrix used for whitening

Return type:

tuple[Tensor, Tensor]

congruence_rectangular(data, weight)

Forward pass of the congruence of a batch of SPD matrices with a (full-rank) rectangular matrix.

\[P' = W^\top P\, W, \qquad W \in \mathbb{R}^{n_{in} \times n_{out}}\]

with \(n_{in} \geq n_{out}\). This is the BiMap layer’s core operation (BiMap): a dimension reduction that keeps the result SPD as long as \(W\) has full column rank.

Parameters:

• data (torch.Tensor of shape (..., n_in, n_in)) – Batch of SPD matrices

• weight (torch.Tensor of shape (n_in, n_out)) – Rectangular matrix (e.g., weights), n_in > n_out is expected

Returns:

data_transformed (torch.Tensor of shape (..., n_out, n_out)) – Transformed batch of SPD matrices

Return type:

Tensor

class CongruenceRectangular(*args, **kwargs)

Congruence of a batch of SPD matrices with a (full-rank) rectangular matrix

static forward(ctx, data, weight)

Forward pass of the congruence of a batch of SPD matrices with a (full-rank) rectangular matrix

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• data (torch.Tensor of shape (..., n_in, n_in)) – Batch of SPD matrices

• weight (torch.Tensor of shape (n_out, n_in)) – Rectangular matrix (e.g., weights), n_in > n_out is expected

Returns:

data_transformed (torch.Tensor of shape (..., n_out, n_out)) – Transformed batch of SPD matrices

Return type:

Tensor

static backward(ctx, grad_output)

Backward pass of the congruence of a batch of SPD matrices with a (full-rank) rectangular matrix

Parameters:

• ctx (torch.autograd.function._ContextMethodMixin) – Context object to retrieve tensors saved during the forward pass

• grad_output (torch.Tensor of shape (..., n_out, n_out)) – Gradient of the loss with respect to the batch of transformed SPD matrices

Returns:

• grad_input_data (torch.Tensor of shape (..., n_in, n_in)) – Gradient of the loss with respect to the input batch of SPD matrices

• grad_input_W (torch.Tensor of shape (n_in, n_out)) – Gradient of the loss with respect to the (full-rank) rectangular matrix W

Return type:

tuple[Tensor, Tensor]