Sparse autoencoders (SAEs) are a popular tool for interpreting hidden activations of neural networks. They learn an over- or equi-dimensional sparse code that approximately reconstructs the original activations.
Implement train_sae(X, W_enc, W_dec, b_enc, lr, l1_coef, num_steps) that trains a single-hidden-layer ReLU sparse autoencoder by full-batch gradient descent and returns the final loss (a single float, not rounded inside the function).
The model is:
The loss to minimize is:
L = mean_over_all_elements( (x_hat - X)^2 ) + l1_coef * mean_over_samples( sum_j z_ij )
That is, the reconstruction term divides by N*D (total number of entries in X) and the sparsity term divides only by N (number of samples).
Shapes:
At every step, compute gradients of L with respect to W_enc, W_dec, b_enc and update them with vanilla SGD using learning rate lr. After num_steps updates (which may be 0), compute and return L on the same batch X with the updated parameters.
Inputs may be passed as lists or numpy arrays; convert internally. Do not modify the inputs in place. Return a Python float.