Layer-sequential unit-variance (LSUV) initialisation starts every layer from an orthonormal matrix and then uses one minibatch to rescale the layers, first to last, so that each layer's pre-activations have unit variance.
Implement lsuv_init(layer_dims, X_batch, act, seed, tol=0.1, max_iter=10) for a bias-free MLP whose layer computes and with = X_batch:
layer_dims: , so has shape . act is "relu", "tanh" or "linear" and is applied after every layer.rng = np.random.default_rng(seed) once; for in order draw G = rng.standard_normal((d_l, d_{l-1})). If , take the reduced QR factorisation and set (orthonormal columns); otherwise factorise and set (orthonormal rows). Treat a sign of as .tol and fewer than max_iter rescalings have been made, set and recompute and . Then feeds the next layer.Return (Ws, variances, n_rescales): the list of final weight matrices, the final of every layer, and how many rescalings each layer needed.