Spooky Sale: 10% off Premium yearly · $118.80 for the first year · until Nov 6Get it

Deep-ML
Learn
Problems1,200+ hands-on ML problemsLearning PathA guided route through the fundamentalsLabsInteractive labs on real techniquesProjectsBuild a GPT, an RL agent, CUDA kernelsCollectionsCurated problem sets and videosMathPen-and-paper math for MLIntuitionInvent the idea behind the formula yourselfPlaylistsYour own problem lists, shareableStudy AssistantAn AI teacher that knows your progress
Interview Prep
Compete
ContestsFresh problem sets, ranked liveLeaderboardWhere you stand, globallyBuddiesSolve with a friend: streaks, a weekly board, chat
Discuss
Deep-Path
Premium

      Machine Learning Problem Collections

      Curated collections of machine learning problems organized by topic. Track your progress through each collection, complete all problems to earn badges, and follow structured learning sequences covering neural networks, linear algebra, optimization, and more.

      Collections
      Attention Is All You Need badge

      Attention Is All You Need

      Vaswani et al.'s 'Attention Is All You Need' (2017), section by section. The attention it replaced, scaled dot-product and multi-head attention, the three ways the model uses attention (masked self-attention and encoder-decoder cross-attention), the encoder and decoder stacks with position-wise feed-forward blocks, residuals and layer normalization, shared embedding and softmax weights with sinusoidal positional encodings, the paper's own 'why self-attention' complexity argument, its training recipe (byte-pair data, Adam with the warmup-then-inverse-square-root schedule, dropout and label smoothing), and its results (beam search with length penalty, BLEU, checkpoint averaging). Math cards derive the numbers the paper states, problems implement each component, and the final section builds the full Transformer from scratch.

      0/50
      0%

      Learning Resource

      Study material to help you master this collection

      Read alongOpen

      Model blueprint

      Transformer

      Attention Is All You Need · Vaswani et al., 2017

      × 6× 6ENCODERDECODERK · VInput EmbeddingPositional Encodingsin / cosMulti-Head Attention8 heads · selfAdd & NormFeed Forward512 → 2048 → 512Add & NormOutput Embeddingshifted rightPositional Encodingsin / cosMasked Self-Attentioncausal maskAdd & NormCross-AttentionQ: decoder · K,V: encoderAdd & NormFeed Forward512 → 2048 → 512Add & NormLinear512 → vocabSoftmaxnext-token probs
      datalearned weightsattentionfixed op
      00/16

      1. Background: Sequence Transduction and Attention (sections 1-2)

      0/2
      • Attention as a Soft Dictionary LookupMath
      • Bahdanau Additive Attention

      2. Scaled Dot-Product Attention (3.2.1)

      0/6
      • Why Scale by sqrt(d_k)Math
      • Simple Self-Attention Without Trainable Weights
      • Scale Attention Scores by sqrt(d_k)
      • Compare Naive vs Stable Softmax for Attention Scores
      • Build Scaled Dot-Product Attention
      • Implement Self-Attention Mechanism

      3. Multi-Head Attention (3.2.2)

      0/7
      • Implement Multi-Head Attention
      • Multi-Head Attention via Head Stacking
      • Multi-Head Attention with Combined QKV Weight Matrix
      • Implement Multi-Head Attention Using Einsum
      • Batched Attention Score Computation for Multiple Heads
      • Implement Multi-Head Self-Attention
      • Design Your Own Attention MechanismLab

      4. Attention in the Model: Masked Self-Attention and Cross-Attention (3.2.3)

      0/6
      • Construct Causal Attention Mask via tril and triu Methods
      • Implement Masked Self-Attention
      • Implement Batched Causal Self-Attention
      • Build Attention Mask from Pad Tokens
      • Causal and Padding Masks with scaled_dot_product_attention
      • Encoder-Decoder Cross-Attention

      5. Encoder and Decoder Stacks: Feed-Forward, Residuals and Layer Normalization (3.1, 3.3)

      0/7
      • Implement a Simple Residual Block with Shortcut Connection
      • Implement Layer Normalization for Sequence Data
      • Implement LayerNorm from Scratch
      • Implement Position-wise Feed-Forward Block with Residual and Dropout
      • Pre-Norm vs Post-Norm Transformer Block
      • Build a Transformer Encoder Layer
      • Implement a Transformer Encoder Block

      6. Embeddings, Shared Softmax Weights and Positional Encoding (3.4-3.5)

      0/7
      • Sinusoidal Positional EncodingsMath
      • Token Embedding Lookup Table
      • Scaled Token Embedding Lookup
      • Implement Weight Tying Between Embedding and LM Head
      • Positional Encoding Calculator
      • Sinusoidal Positional Encoding
      • Learned Positional Embeddings

      7. Why Self-Attention (section 4)

      0/1
      • Multi-Head Attention: Parameters and FLOPs per LayerMath

      8. Training: Data, Optimizer, Schedule and Regularization (section 5)

      0/8
      • The Transformer Learning-Rate ScheduleMath
      • Label SmoothingMath
      • Byte Pair Encoding (BPE) Tokenizer
      • Implement Adam Optimization Algorithm
      • Transformer Warmup Learning-Rate Schedule
      • Dropout Layer
      • Apply Dropout to Attention Weights
      • Implement Label Smoothing for Multi-Class Cross-Entropy

      9. Results: Decoding, BLEU and Checkpoint Averaging (section 6)

      0/5
      • Transformer Parameter Count: Base and BigMath
      • Beam Search with Length Penalty
      • Beam Search Decoding
      • BLEU Score for Text Generation
      • Checkpoint Averaging (Polyak Averaging)

      10. Build the Transformer

      0/1
      • Attention Is All You Need: Build the Transformer From ScratchProject