Mind map of But what is a neural network?

YouTube · 18:40 · 101 nodes

What you add

But what is a neural network? | Deep learning chapter 1

YouTube · 18:40Pro

18:40 · ~3,400 words becomes 101 nodes

But what is a neural network?

3Blue1Brown on YouTube

Try 1 YouTube video up to 20 minutes without signing up (no credits). A free account adds 1 more try (uses credits). Unlimited YouTube input is on Pro and Max. See pricing

View source

Interactive mind map

Drag to move, click a node to fold it.

Open in editorFree account

Try it with your own contentFree account

Summary

In the first chapter of 3Blue1Brown's deep learning series, Grant Sanderson explains what a neural network is by building one that reads handwritten digits. The network is a stack of layers of neurons, each holding an activation between 0 and 1: 784 input neurons for the pixels of a 28×28 image, two hidden layers of 16 neurons, and 10 output neurons for the digits. Activations in one layer determine the next through weighted sums plus a bias, squished into range by a sigmoid (or, in modern practice, ReLU). The hope is that hidden layers learn to detect edges, then loops and lines, then whole digits. Counting the weights and biases gives about 13,000 parameters, and "learning" means finding values for them, which is the subject of the next chapter.

Key takeaways

A neuron is just a number between 0 and 1; a layer is a list of them, and the network is layers connected by weights.
Digit recognition uses 784 input neurons (28×28 pixels), two hidden layers of 16, and 10 outputs.
Each activation is sigmoid(weighted sum of the previous layer + bias); ReLU is the modern replacement for sigmoid.
Hidden layers are hoped to build up features: edges, then loops and lines, then digits.
This small network has about 13,000 weights and biases; training is the search for their values.

All 101 nodes

The full mind map as an outline.

Understanding Neural Networks: Structure and Function

Neural Networks: A Primer
- Human Brain Analogy
  - Effortless recognition of complex patterns (e.g., handwritten digits)
  - Visual cortex processes diverse inputs as same idea
- The Challenge of Programming AI
  - Simple concept (recognizing a '3') becomes difficult for traditional programming
  - Highlights the need for Machine Learning and Neural Networks
- Relevance and Importance
  - Present and future of technology
What is a Neural Network?
- Inspired by the Brain
  - Neurons: Units holding numbers between 0 and 1
  - Activation: The numerical value held by a neuron
- Layered Structure
  - Input Layer
    - Represents pixels of the input image (e.g., 28x28 = 784 neurons)
    - Activation represents grayscale value (0 for black, 1 for white)
  - Output Layer
    - 10 neurons, each representing a digit (0-9)
    - Activation indicates confidence in the digit recognition
  - Hidden Layers
    - Intermediate layers between input and output
    - Responsible for complex pattern recognition
    - Example: Two hidden layers with 16 neurons each (arbitrary choice)
- Information Flow
  - Activations in one layer determine activations in the next
  - Loosely analogous to biological neuron firing
The Process of Recognition
- Trained Network Behavior
  - Input image activates input layer neurons
  - Pattern of activations propagates through layers
  - Output layer's brightest neuron indicates the recognized digit
- Hope for Hidden Layers: Hierarchical Feature Detection
  - Recognizing digits by combining components
  - Example: '9' has a loop and a line
  - Potential role of hidden layers:
    - Second-to-last layer: Detects subcomponents (e.g., a loop)
    - Second layer: Detects fundamental features (e.g., edges)
  - Generalizability: Useful for other image recognition tasks
  - Broader applications: Speech parsing, abstract thought
How Activations Propagate: The Math
- Core Mechanism: Combining Layer Activations
  - Goal: Combine pixels into edges, edges into patterns, patterns into digits
- Detecting Patterns (e.g., an edge in a specific region)
  - Parameters: Weights and Biases
    - Weights: Numbers associated with connections between neurons
      - Represent the importance of a connection
      - Organized into a grid (green for positive, red for negative)
    - Weighted Sum: Sum of activations from the previous layer multiplied by their respective weights
    - Bias: An additional number added to the weighted sum
      - Represents a threshold for neuron activation
- Squishing Output: Activation Functions
  - Goal: Output activation between 0 and 1
  - Sigmoid Function (Logistic Curve)
    - Squishes any real number into the [0, 1] range
    - Input 0 maps to 0.5, very negative to 0, very positive to 1
  - Output Activation: A measure of how positive the weighted sum is, after applying the sigmoid
Network Complexity
- Number of Parameters
  - Weights and biases for each connection
  - Example: 784 neurons * 16 neurons/layer * 2 hidden layers + output layer connections
  - Total: Approximately 13,000 weights and biases
- "Learning" Defined
  - Finding the correct settings for all weights and biases
  - Enables the network to solve the intended problem
- Manual Tuning (Thought Experiment)
  - Satisfying to understand what parameters mean
  - Aids in debugging and improving the network
Notation for Connections
- Vector Representation
  - Activations of a layer organized into a column vector
- Matrix Representation
  - Weights organized into a matrix
  - Matrix-vector product computes weighted sums efficiently
- Bias Vector
  - Biases organized into a vector and added to the matrix-vector product
- Applying Activation Function
  - Applied element-wise to the resulting vector
- Compact Expression
  - sigmoid(W * a + b)
  - W: Weight matrix, a: Activation vector, b: Bias vector
The Network as a Function
- Neurons as Functions
  - Take previous layer outputs, produce a value between 0 and 1
- Entire Network as a Function
  - Input: 784 pixel values
  - Output: 10 digit probabilities
  - Complex function with ~13,000 parameters
Future Steps: Learning
- How the network learns appropriate weights and biases from data
- Deeper dive into what the specific network is doing
Discussion on Activation Functions
- Sigmoid Function
  - Early choice, inspired by biological analogy
  - Can be difficult to train ("old school")
- ReLU (Rectified Linear Unit)
  - Modern choice, widely used
  - max(0, a) where a is the weighted sum
  - Easier to train, especially for deep networks
  - Simplification of biological activation (either off or on)