Implement a complete Transformer Encoder Layer forward pass using only NumPy. The encoder layer follows the architecture from "Attention Is All You Need" (Vaswani et al., 2017) and consists of four main sub-operations applied in sequence:
Multi-Head Self-Attention: Project the input into queries, keys, and values using weight matrices, split into multiple attention heads, compute scaled dot-product attention for each head, concatenate the results, and apply an output projection.
Add & Norm (first): Apply a residual connection (add the sub-layer input to its output) followed by layer normalization.
Position-wise Feed-Forward Network: Apply two linear transformations with a ReLU activation in between.
Add & Norm (second): Apply another residual connection followed by layer normalization.
Your function should accept:
X: input tensor of shape (batch_size, seq_len, d_model)weights: a dictionary containing all layer parameters:
'W_q', 'W_k', 'W_v': query, key, value projection matrices, each of shape (d_model, d_model)'W_o': output projection matrix of shape (d_model, d_model)'W1', 'b1': first FFN weight (d_model, d_ff) and bias (d_ff,)'W2', 'b2': second FFN weight (d_ff, d_model) and bias (d_model,)'gamma1', 'beta1': first layer norm parameters, each of shape (d_model,)'gamma2', 'beta2': second layer norm parameters, each of shape (d_model,)num_heads: number of attention heads (must evenly divide d_model)eps: layer normalization epsilon (default 1e-5)Return the output tensor of shape (batch_size, seq_len, d_model).
Use the standard softmax with numerical stability (subtract the max before exponentiating). Layer normalization should normalize over the last dimension using population variance (denominator N, not N-1).