Module 3

Convolutional & Recurrent Networks

Context-setting module. CNNs, RNNs, and the vanishing gradient problem that motivates LSTMs and Transformers.

This module is primarily context-setting. We implement these architectures at a high level to understand their role in the evolution toward Transformers — not to use them in production.

Learning Objectives

  • Understand convolutional layers and parameter sharing
  • Understand recurrent networks and their limitations
  • Recognise the vanishing gradient problem
  • Know when to use each architecture

3.1 Convolutional Neural Networks

Key Concepts

  • Convolution operation (not circular convolution)
  • Pooling layers (max pool, average pool)
  • Parameter sharing and translation invariance
  • Success on vision tasks
CNNs show how inductive biases (local connectivity, weight sharing) help learning. This concept of built-in structure persists in modern architectures.
Haskell — 2D convolution sketch
-- Convolution: element-wise multiply filter with local patch, sum
convolve2D :: Matrix Double -> Matrix Double -> Matrix Double
convolve2D input filter =
  let patchesAndFiltered = [convolvePatches input filter]
  in reshape patchesAndFiltered

-- Key insight: much smaller model than fully connected
-- For 28x28 image: FC layer  = 784 × 784 = 614k params
-- Conv layer with 32 filters of 5×5 = 32 × (5×5 + 1) = 832 params

3.2 Recurrent Neural Networks

Key Concepts

  • Recurrent connections and weight sharing across time
  • Hidden state as memory
  • Backpropagation through time (BPTT)
  • The vanishing gradient problem
RNNs handle sequences but fail at long-term dependencies. This problem motivates LSTMs (Module 4) and ultimately Transformers (Module 6).
Haskell — RNN forward pass
data RNNState = RNNState
  { weightsInput  :: Matrix Double
  , weightsHidden :: Matrix Double
  , bias          :: Vector Double
  , hiddenSize    :: Int
  }

-- Forward pass for one timestep
rnnStep :: RNNState -> Vector Double -> Vector Double -> Vector Double
rnnStep state input prevHidden =
  let combined = matmul (weightsInput state) input
               + matmul (weightsHidden state) prevHidden
               + bias state
  in fmap tanh combined

-- Process a full sequence
rnnForward :: RNNState -> [Vector Double] -> [Vector Double]
rnnForward state inputs =
  let initialHidden = zeroVector (hiddenSize state)
      (_, outputs)  = foldl processStep (initialHidden, []) inputs
  in reverse outputs
  where
    processStep (hidden, acc) input =
      let newHidden = rnnStep state input hidden
      in (newHidden, newHidden : acc)

The Vanishing Gradient Problem

When backpropagating through many timesteps, gradients either vanish (multiply by values < 1 repeatedly) or explode (multiply by values > 1). This makes learning long-term dependencies nearly impossible. The cell state in LSTMs was specifically designed to solve this.

Reading & Viewing

CNNs

Gradient-Based Learning Applied to Document Recognition (1998) LeCun, Bottou, Bengio, Haffner — LeNet-5 yann.lecun.com — lecun-98.pdf
Stanford CS231n — Lecture 5 (CNNs) Andrej Karpathy YouTube

RNNs & vanishing gradients

On the Difficulty of Training Recurrent Neural Networks (2001) Hochreiter, Bengio, Frasconi, Schmidhuber Rigorous analysis of the vanishing gradient problem. arxiv.org/abs/1211.1541
Sequence to Sequence Learning with Neural Networks (2014) Sutskever, Vinyals, Le arxiv.org/abs/1409.3215

Blog

Understanding LSTM Networks Chris Olah Read this before Module 4 — sets up the intuition perfectly. colah.github.io/posts/2015-08-Understanding-LSTMs/