Mind map of Attention Is All You Need

PDF · 15 pages · 77 nodes

What you add

1706.03762v7.pdf

PDF · 15 pages

15 pages · ~6,100 words becomes 77 nodes

Attention Is All You Need Abstract The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Attention Is All You Need

Vaswani et al., arXiv:1706.03762

View source

Interactive mind map

Drag to move, click a node to fold it.

Open in editorFree account

Try with your own fileNo sign-up, up to 4 MB

Summary

Attention Is All You Need (Vaswani et al., 2017) introduces the Transformer, a sequence transduction model built entirely on attention mechanisms, with no recurrence or convolutions. An encoder and a decoder, each a stack of six identical layers, use multi-head self-attention and position-wise feed-forward networks, with residual connections and layer normalization around every sub-layer. Because the model has no notion of order on its own, sinusoidal positional encodings are added to the input embeddings. Self-attention gives constant-length paths between any two positions, which makes long-range dependencies easier to learn and the whole model far more parallelizable than RNNs. On WMT 2014 the Transformer reached 28.4 BLEU on English-to-German and 41.8 BLEU on English-to-French, beating previous state-of-the-art results at a fraction of the training cost, and it generalized to English constituency parsing.

Key takeaways

The Transformer replaces recurrence and convolution with attention alone; the encoder and decoder are each six identical layers.
Scaled dot-product attention, softmax(QKᵀ/√dk)V, and multi-head attention let the model attend to different representation subspaces in parallel.
Sinusoidal positional encodings inject word order, since attention by itself is order-agnostic.
Self-attention needs O(1) sequential operations and gives O(1) path length between positions, versus O(n) for RNNs.
28.4 BLEU on WMT 2014 English-German and 41.8 on English-French, trained in 3.5 days on 8 P100 GPUs.

All 77 nodes

The full mind map as an outline.

Attention Is All You Need - Overview

The Transformer Model
- Core Idea: Solely based on attention mechanisms, eschewing recurrence and convolutions.
- Architecture: Encoder-Decoder structure.
- Key Components:
  - Encoder Stack
    - N = 6 identical layers
    - Sub-layers:
      - Multi-head self-attention
      - Position-wise feed-forward network
    - Residual connections and Layer Normalization
  - Decoder Stack
    - N = 6 identical layers
    - Sub-layers:
      - Masked multi-head self-attention
      - Multi-head attention (encoder-decoder attention)
      - Position-wise feed-forward network
    - Residual connections and Layer Normalization
- Attention Mechanism
  - Scaled Dot-Product Attention
    - Formula: softmax(QK^T / sqrt(dk)) V
    - Purpose: Efficient computation, prevents large gradients with scaling.
  - Multi-Head Attention
    - Mechanism: Projects Q, K, V multiple times, performs attention in parallel, concatenates and projects.
    - Benefits: Allows attending to information from different representation subspaces.
    - Applications in Transformer:
      - Encoder-Decoder Attention
      - Encoder Self-Attention
      - Decoder Self-Attention (masked)
- Position-wise Feed-Forward Networks
  - Structure: Two linear transformations with ReLU activation.
  - Application: Applied to each position separately and identically.
- Embeddings and Softmax
  - Input/Output Embeddings: Learned embeddings to convert tokens to vectors.
  - Shared Weights: Same weight matrix for embedding layers and pre-softmax linear transformation.
- Positional Encoding
  - Purpose: Injects sequence order information since no recurrence/convolution.
  - Method: Sine and cosine functions of different frequencies.
  - Hypothesis: Allows learning relative positions.
Advantages
- Superior Quality: Achieves state-of-the-art results in machine translation.
- Parallelizability: Significantly more parallelizable than recurrent models.
- Training Time: Requires significantly less time to train.
Performance
- Machine Translation:
  - WMT 2014 English-to-German: 28.4 BLEU (superior to ensembles).
  - WMT 2014 English-to-French: 41.8 BLEU (state-of-the-art single-model).
- English Constituency Parsing: Generalizes well to other tasks.
Why Self-Attention (Comparison to RNN/Convolutional)
- Computational Complexity: O(n^2 * d) for Self-Attention vs. O(n * d^2) for RNN.
- Parallelizability: O(1) sequential operations for Self-Attention vs. O(n) for RNN.
- Path Length: O(1) for Self-Attention vs. O(n) for RNN, facilitating learning long-range dependencies.
Training Details
- Data: WMT 2014 English-German and English-French datasets.
- Batching: By approximate sequence length.
- Hardware: 8 NVIDIA P100 GPUs.
- Schedule: 100K steps (12 hours) for base, 300K steps (3.5 days) for big models.
- Optimizer: Adam (with specific learning rate schedule).
- Regularization:
  - Residual Dropout
  - Label Smoothing
Model Variations and Experiments
- Impact of Attention Heads and Dimensions
- Importance of Key Dimension (dk)
- Effect of Model Size and Dropout
- Positional Encoding: Sinusoidal vs. Learned Embeddings.
Conclusion and Future Work
- Demonstrated effectiveness of attention-only models.
- Future directions:
  - Extending to other modalities (images, audio, video).
  - Investigating local, restricted attention.
  - Making generation less sequential.
Authorship and Contributions
- Equal contribution from all authors.
- Specific contributions mentioned for key ideas and implementation.
Permissions
- Google grants permission for reproduction of tables/figures in journalistic or scholarly works with attribution.