Implement TransformerBlock(d_model, num_heads, d_ff, dropout=0.1) as an nn.Module. The block must contain, in this order:
self.norm1: nn.LayerNorm(d_model)self.attn: nn.MultiheadAttention(d_model, num_heads, dropout=dropout, batch_first=True)self.norm2: nn.LayerNorm(d_model)self.mlp: an nn.Sequential of Linear(d_model, d_ff) -> GELU -> Linear(d_ff, d_model)self.dropout: nn.Dropout(dropout), used on the output of attention and the output of the MLP before the residual adds.Use the pre-LN pattern: normalize first, then apply the sublayer, then add the residual. Concretely, forward(x) should be equivalent to two sublayer applications:
and return the result.
nn.MultiheadAttention with batch_first=True returns a tuple (out, attn_weights). You only need the output.