Build the Sinusoidal Position Table
Implement sinusoidal_positional_encoding(seq_len, d_model) that returns a tensor of shape (seq_len, d_model) whose entries follow the standard transformer recipe:
- Even columns (2i) use sine: PEp,2i=sin(p/100002i/dmodel).
- Odd columns (2i+1) use cosine: PEp,2i+1=cos(p/100002i/dmodel).
You may assume d_model is even.
Hints
- Build a
position column vector of shape (seq_len, 1) and a div_term row vector of shape (d_model // 2,); the outer product gives you the matrix of angles in shape (seq_len, d_model // 2).
- Numerical tip: compute the division term in log space to avoid overflow at large dmodel: 1/100002i/d=exp(−log(10000)⋅2i/d).
- Fill even columns with
sin(angles) and odd columns with cos(angles).