Mind map of Deep Residual Learning for Image Recognition

PDF · 12 pages · 67 nodes

What you add

1512.03385v1.pdf

PDF · 12 pages

12 pages · ~9,800 words becomes 67 nodes

Deep Residual Learning for Image Recognition Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun Microsoft Research {kahe, v-xiangz, v-shren, jiansun}@microsoft.com Abstract Deeper neural networks are more difficult to train. We

Deep Residual Learning for Image Recognition

He et al., arXiv:1512.03385

View source

Interactive mind map

Drag to move, click a node to fold it.

Open in editorFree account

Try with your own file

All 67 nodes

The full mind map as an outline.

Deep Residual Learning for Image Recognition

Introduction
- Importance of network depth for visual recognition tasks
- Breakthroughs by deep convolutional neural networks
- Previous work on very deep models (VGG, GoogLeNet)
- Motivation: Can deeper networks always improve accuracy?
Core Problem: Degradation
- Difficulty in training deeper networks
- Vanishing/exploding gradients (largely addressed)
- The degradation problem: Accuracy saturates and then degrades with increased depth
- Not due to overfitting; higher training error in deeper "plain" networks
- Implication: Current solvers struggle to optimize deep plain networks effectively
Proposed Solution: Residual Learning
- Reformulating layers to learn residual functions F(x) := H(x) - x
- Original mapping H(x) becomes F(x) + x
- Hypothesis: Easier to learn residual mapping than original unreferenced mapping
- Extreme case: Identity mapping is optimal, residual function can be driven to zero
- Analogy: Preconditioning the problem, easier to learn perturbations around an identity
Key Component: Shortcut Connections
- Realization of F(x) + x through shortcut connections
- Identity mapping shortcuts: add no parameters or computational complexity
- Element-wise addition of shortcut output and stacked layer output
- Flexibility: Can use linear projection (Ws) to match dimensions if needed
- Bottleneck architectures: Use 1x1 convolutions to reduce/restore dimensions for efficiency
- Parameter-free identity shortcuts are crucial for bottleneck designs
Network Architectures
- Plain Networks: Inspired by VGG, with 3x3 convs, doubling filters with halved feature map size
- Residual Networks (ResNets): Inserting shortcut connections into plain networks
- Identity vs. Projection Shortcuts:
  - Identity: Parameter-free, preferred when dimensions match
  - Projection (Ws): Used to match dimensions, adds parameters and complexity
  - Zero-padding: Option for dimension matching without extra parameters
- Bottleneck Designs: 1x1, 3x3, 1x1 convs for deeper networks (ResNet-50/101/152)
- Example Architectures: VGG-19, 34-layer plain, 34-layer residual
Experimental Results: ImageNet
- Evaluation on ImageNet 2012 classification dataset
- Plain Networks: 18-layer and 34-layer show degradation (deeper has higher training error)
- Residual Networks (ResNets):
  - 34-layer ResNet significantly outperforms 18-layer ResNet
  - Addresses degradation problem, gains accuracy from increased depth
  - Outperforms plain counterparts substantially
- Deeper ResNets (50, 101, 152 layers):
  - Achieve state-of-the-art results
  - 152-layer ResNet is the deepest network at the time, with lower complexity than VGG
  - Ensemble achieves 3.57% top-5 error, winning ILSVRC 2015
- Identity vs. Projection Shortcuts: Minimal difference, identity is more economical
Experimental Results: CIFAR-10
- Analysis on CIFAR-10 dataset with deeper networks
- Plain Nets: Exhibit degradation similar to ImageNet
- ResNets: Overcome optimization difficulty, gain accuracy with depth
- Explored up to 110 layers and 1202 layers
- 1202-layer network shows excellent training performance but worse test error (overfitting)
- Analysis of Layer Responses: ResNets show smaller response magnitudes, supporting residual function hypothesis
Applications: Object Detection & Localization
- Good generalization to other recognition tasks
- Object Detection (PASCAL VOC, MS COCO):
  - Replacing VGG-16 with ResNet-101 yields significant mAP improvements (e.g., 28% relative on COCO)
  - Won 1st place in ILSVRC & COCO 2015 detection tasks
- ImageNet Localization:
  - Novel per-class RPN and R-CNN framework
  - Significantly reduced localization error compared to prior methods
  - Won 1st place in ILSVRC 2015 localization task
Related Work & Future Directions
- Residual Representations (VLAD, Fisher Vector)
- Shortcut Connections in prior work (MLPs, auxiliary classifiers, Highway Networks)
- Differences from Highway Networks (parameter-free vs. gated shortcuts)
- Future work: Exploring stronger regularization for extremely deep models, applicability to other domains