Loss-of-Plasticity Diagnostics: Dormant Units and Effective Rank

EasyDeep Learning

Measure Whether a Network Can Still Learn

A network trained on a long sequence of tasks can lose the ability to learn new ones. Three diagnostics track it: the fraction of dormant units, the effective rank of a layer's features, and the growth of the weight norm. Implement them on a matrix of hidden activations.

dormant_ratio(H, tau)

H is an (N, U) array of post-activation values of one layer over $N$ inputs. The score of unit $u$ is its mean absolute activation divided by the mean of those means over all units: $s_u = \frac{\frac{1}{N}\sum_n |H_{n,u}|}{\frac{1}{U}\sum_{u'} \frac{1}{N}\sum_n |H_{n,u'}|}$ . If every unit has mean absolute activation zero, every score is 0.
A unit is dormant when $s_u \le \tau$ . Return (ratio, dormant_indices): the fraction of dormant units as a Python float and the sorted list of their indices.

effective_rank(H, eps=1e-12)

Centre the columns of H, take the singular values $\sigma_i$ of the centred matrix, normalise them to $p_i = \sigma_i / \sum_j \sigma_j$ (ignoring values below eps), and return $\exp\!\left(-\sum_i p_i \log p_i\right)$ as a Python float. If all singular values are below eps, return 0.0.

plasticity_report(H, W, tau, decimals=4)

W is the layer's weight matrix of any shape. Return a dict with keys "dormant_ratio", "effective_rank", "weight_norm" (the Frobenius norm of W), rounded to decimals.

Hints

np.abs(H).mean(axis=0) gives one number per unit; the normaliser is its mean.
np.linalg.svd(Hc, compute_uv=False) returns the singular values directly.
Effective rank of a matrix whose columns are all identical is exactly one; of orthogonal columns with equal norms it equals the number of columns.

Example:

Input:

```python
import numpy as np
H = np.array([[1.0, 2.0, 0.0, 1.0], [0.0, 2.0, 0.0, 0.0], [1.0, 0.0, 0.0, 1.0], [2.0, 4.0, 0.0, 2.0]])
print(dormant_ratio(H, 0.1))
print(round(effective_rank(H), 4))
print(plasticity_report(H, np.array([[3.0, 4.0]]), 0.1))
```

Output:

(0.25, [2])
1.8998
{'dormant_ratio': 0.25, 'effective_rank': 1.8998, 'weight_norm': 5.0}

Reasoning:

The third unit is zero on every input, so its score is zero and it is dormant; the other three have scores above the threshold. After centring, the four columns span only two independent directions with unequal singular values, so the effective rank lies between one and two.

Learn About topic

Contributors:

Deep-ML

Contribute

Measure Whether a Network Can Still Learn

dormant_ratio(H, tau)

H is an (N, U) array of post-activation values of one layer over

N

inputs. The score of unit

u

is its mean absolute activation divided by the mean of those means over all units:

s_u = \frac{\frac{1}{N}\sum_n |H_{n,u}|}{\frac{1}{U}\sum_{u'} \frac{1}{N}\sum_n |H_{n,u'}|}

. If every unit has mean absolute activation zero, every score is 0.

A unit is dormant when

s_u \le \tau

. Return (ratio, dormant_indices): the fraction of dormant units as a Python float and the sorted list of their indices.

effective_rank(H, eps=1e-12)

Centre the columns of H, take the singular values

\sigma_i

of the centred matrix, normalise them to

p_i = \sigma_i / \sum_j \sigma_j

(ignoring values below eps), and return

\exp\!\left(-\sum_i p_i \log p_i\right)

as a Python float. If all singular values are below eps, return 0.0.

plasticity_report(H, W, tau, decimals=4)

W is the layer's weight matrix of any shape. Return a dict with keys "dormant_ratio", "effective_rank", "weight_norm" (the Frobenius norm of W), rounded to decimals.

Hints

np.abs(H).mean(axis=0) gives one number per unit; the normaliser is its mean.

np.linalg.svd(Hc, compute_uv=False) returns the singular values directly.

Effective rank of a matrix whose columns are all identical is exactly one; of orthogonal columns with equal norms it equals the number of columns.