Mind map of Attention Is All You Need

PDF · 15 pages · 77 nodes

What you add

1706.03762v7.pdf

PDF · 15 pages

15 pages · ~6,100 words becomes 77 nodes

Excerpt from the original: Provided proper attribution is provided, Google hereby grants permission to reproduce the tables and figures in this paper solely for use in journalistic or scholarly works. Attention Is All You Need Ashish Vaswani∗ Google Brain

Attention Is All You Need

Vaswani et al., arXiv:1706.03762

View source

Interactive mind map

Mind map preview: Attention Is All You Need - OverviewLoading the interactive mind map…

Drag to move, click a node to fold it.

Open in editorFree account
Try with your own file

All 77 nodes

The full mind map as an outline.

Attention Is All You Need - Overview

  • The Transformer Model
    • Core Idea: Solely based on attention mechanisms, eschewing recurrence and convolutions.
    • Architecture: Encoder-Decoder structure.
    • Key Components:
      • Encoder Stack
        • N = 6 identical layers
        • Sub-layers:
          • Multi-head self-attention
          • Position-wise feed-forward network
        • Residual connections and Layer Normalization
      • Decoder Stack
        • N = 6 identical layers
        • Sub-layers:
          • Masked multi-head self-attention
          • Multi-head attention (encoder-decoder attention)
          • Position-wise feed-forward network
        • Residual connections and Layer Normalization
    • Attention Mechanism
      • Scaled Dot-Product Attention
        • Formula: softmax(QK^T / sqrt(dk)) V
        • Purpose: Efficient computation, prevents large gradients with scaling.
      • Multi-Head Attention
        • Mechanism: Projects Q, K, V multiple times, performs attention in parallel, concatenates and projects.
        • Benefits: Allows attending to information from different representation subspaces.
        • Applications in Transformer:
          • Encoder-Decoder Attention
          • Encoder Self-Attention
          • Decoder Self-Attention (masked)
    • Position-wise Feed-Forward Networks
      • Structure: Two linear transformations with ReLU activation.
      • Application: Applied to each position separately and identically.
    • Embeddings and Softmax
      • Input/Output Embeddings: Learned embeddings to convert tokens to vectors.
      • Shared Weights: Same weight matrix for embedding layers and pre-softmax linear transformation.
    • Positional Encoding
      • Purpose: Injects sequence order information since no recurrence/convolution.
      • Method: Sine and cosine functions of different frequencies.
      • Hypothesis: Allows learning relative positions.
  • Advantages
    • Superior Quality: Achieves state-of-the-art results in machine translation.
    • Parallelizability: Significantly more parallelizable than recurrent models.
    • Training Time: Requires significantly less time to train.
  • Performance
    • Machine Translation:
      • WMT 2014 English-to-German: 28.4 BLEU (superior to ensembles).
      • WMT 2014 English-to-French: 41.8 BLEU (state-of-the-art single-model).
    • English Constituency Parsing: Generalizes well to other tasks.
  • Why Self-Attention (Comparison to RNN/Convolutional)
    • Computational Complexity: O(n^2 * d) for Self-Attention vs. O(n * d^2) for RNN.
    • Parallelizability: O(1) sequential operations for Self-Attention vs. O(n) for RNN.
    • Path Length: O(1) for Self-Attention vs. O(n) for RNN, facilitating learning long-range dependencies.
  • Training Details
    • Data: WMT 2014 English-German and English-French datasets.
    • Batching: By approximate sequence length.
    • Hardware: 8 NVIDIA P100 GPUs.
    • Schedule: 100K steps (12 hours) for base, 300K steps (3.5 days) for big models.
    • Optimizer: Adam (with specific learning rate schedule).
    • Regularization:
      • Residual Dropout
      • Label Smoothing
  • Model Variations and Experiments
    • Impact of Attention Heads and Dimensions
    • Importance of Key Dimension (dk)
    • Effect of Model Size and Dropout
    • Positional Encoding: Sinusoidal vs. Learned Embeddings.
  • Conclusion and Future Work
    • Demonstrated effectiveness of attention-only models.
    • Future directions:
      • Extending to other modalities (images, audio, video).
      • Investigating local, restricted attention.
      • Making generation less sequential.
  • Authorship and Contributions
    • Equal contribution from all authors.
    • Specific contributions mentioned for key ideas and implementation.
  • Permissions
    • Google grants permission for reproduction of tables/figures in journalistic or scholarly works with attribution.
    Mind map of Attention Is All You Need — made with MindLM