Implement a ReLU sparse autoencoder with an penalty, its exact gradient step, and the diagnostics used to judge a dictionary: , explained variance, dead and dense latents.
sae_forward(X, W_enc, b_enc, W_dec, b_dec)
X (B, d), W_enc (d, n), b_enc (n,), W_dec (n, d), b_dec (d,). pre = X @ W_enc + b_enc, Z = max(pre, 0), Xhat = Z @ W_dec + b_dec. Return (pre, Z, Xhat).sae_loss_and_grads(X, W_enc, b_enc, W_dec, b_dec, lam)
(loss, grads) with grads a dict of exact gradients for the four parameters (keys "W_enc", "b_enc", "W_dec", "b_dec").sae_step(params, X, lam, lr)
params is a dict with the four arrays. Take one gradient step and then renormalise every row of W_dec to unit norm. Return a new dict.sae_diagnostics(Z, X, Xhat, dense_thresh=0.5, decimals=4)
"l0" (mean number of active latents per row), "explained_variance" ( with the column mean), "dead_frac" (fraction of latents active on no row), "dense_frac" (fraction active on more than dense_thresh of rows), rounded.lam / B; it is masked by pre > 0 like the reconstruction gradient.