Implement a causal multi-head self-attention layer that uses a single combined weight matrix to produce queries, keys, and values in one matrix multiplication (instead of three separate projections). Splitting Q/K/V from one fused projection is a common optimization that reduces overhead and improves memory locality.
Given:
x: input tensor of shape (batch, num_tokens, d_in)W_qkv: combined projection matrix of shape (d_in, 3 * d_out) -- the first d_out output columns produce queries, the next d_out produce keys, and the last d_out produce valuesW_out: output projection matrix of shape (d_out, d_out)num_heads: number of attention heads. d_out is divisible by num_heads and head_dim = d_out // num_headsYour function must:
x to a combined Q/K/V tensor of shape (batch, num_tokens, 3 * d_out) using W_qkv (no bias).(batch, num_heads, num_tokens, head_dim) for queries, keys, and values.1 / sqrt(head_dim).i cannot attend to positions j > i (use -inf before softmax).(batch, num_tokens, d_out).W_out and return the result as a nested Python list.No dropout is applied. Use only NumPy and pure Python.