On the way back up, an H-Net does not trust each chunk equally. The smoothing module runs an exponential moving average over the main network's outputs, weighted by the router's kept probabilities:
Implement the recursion and its unrolled form.
ema_smooth(z, P, mask=None)
z float (B, M, D), P float (B, M), mask optional bool (B, M) with True on real positions. Padding, if any, is always at the end of a row.zbar (B, M, D) from the recursion above, starting each row from 0.zbar must be 0.0.z and P. Do not modify the inputs in place.ema_weights(P)
P is a 1-D float tensor of length M.W of shape (M, M) such that for scalar chunk values. It is lower triangular, withand 0.0 above the diagonal.