Building Your First Neural Network

2.1 The Perceptron

Key Concepts

• Single neuron with linear activation
• Perceptron learning rule
• Limitations and the XOR problem

data Neuron = Neuron
  { weights :: Vector Double
  , bias    :: Double
  }

-- Forward pass (linear, no activation yet)
forward :: Neuron -> Vector Double -> Double
forward n x = dot (weights n) x + bias n

-- Prediction (threshold at 0 for binary classification)
predict :: Neuron -> Vector Double -> Bool
predict n x = forward n x > 0

Extra reading

Perceptrons by Minsky and Papert (1969) — the classic text that shows the mathematical limitations of single-layer networks.

2.2 Multilayer Perceptron (MLP)

Key Concepts

• Stacking layers and nonlinearity
• Activation functions: sigmoid, tanh, ReLU
• Solving XOR with hidden layers
• Universal approximation theorem

data Layer = Layer
  { weights    :: Matrix Double
  , bias       :: Vector Double
  , activation :: Activation
  }

data MLP = MLP [Layer]

data Activation = Sigmoid | Tanh | ReLU
  deriving (Show, Eq)

-- Forward pass through network
forwardMLP :: MLP -> Vector Double -> Vector Double
forwardMLP (MLP layers) input = foldl forwardLayer input layers
  where
    forwardLayer x layer =
      activateLayer (activation layer)
                    (matmul (weights layer) x + bias layer)

-- Activation functions
sigmoid :: Double -> Double
sigmoid x = 1 / (1 + exp (-x))

relu :: Double -> Double
relu x = max 0 x

activateLayer :: Activation -> Vector Double -> Vector Double
activateLayer Sigmoid = fmap sigmoid
activateLayer Tanh    = fmap tanh
activateLayer ReLU    = fmap relu

Implementation Task 2.2 — Backpropagation

This is complex but critical. Key steps:

• Compute derivatives through activation functions
• Apply chain rule through multiple layers
• Accumulate gradients
• Numerical gradient checking (sanity-check your implementation)

data ComputeContext = ComputeContext
  { layerInputs  :: [Vector Double]
  , layerOutputs :: [Vector Double]
  , parameters   :: MLP
  }

-- Store forward values needed for backward pass
computeContext :: MLP -> Vector Double -> ComputeContext

-- Backprop through one layer
backpropLayer
  :: Layer
  -> Vector Double          -- upstream gradient
  -> Vector Double          -- layer input
  -> (Gradient, Vector Double)   -- (gradient w.r.t. params, gradient for previous layer)
backpropLayer layer upstreamGrad layerInput =
  let activationGrad = computeActivationGradient (activation layer)
                         (matmul (weights layer) layerInput)
      downstreamGrad = elementwiseMultiply upstreamGrad activationGrad
      weightGrad     = outer downstreamGrad layerInput
      biasGrad       = downstreamGrad
      previousGrad   = matmul (transpose (weights layer)) downstreamGrad
  in ((weightGrad, biasGrad), previousGrad)

-- Verify analytical gradients against numerical finite differences
numericalGradient :: (Vector Double -> Double) -> Vector Double -> Double -> Vector Double
numericalGradient f x epsilon =
  let unitVectors = [[if i == j then 1 else 0 | j <- [1..length x]] | i <- [1..length x]]
  in vector [((f (x + epsilon * e)) - (f (x - epsilon * e))) / (2 * epsilon)
             | e <- unitVectors]

-- Relative error should be < 1e-5

2.3 Training on XOR

xorData :: [(Vector Double, Double)]
xorData = [([0, 0], 0), ([0, 1], 1), ([1, 0], 1), ([1, 1], 0)]

-- Training loop for one epoch
trainEpoch :: MLP -> Double -> [(Vector Double, Double)] -> MLP
trainEpoch network learningRate examples =
  foldl updateNetwork network examples
  where
    updateNetwork net (input, target) =
      let ctx    = computeContext net input
          output = last (layerOutputs ctx)
          loss   = meanSquaredError output [target]
          grads  = backprop net ctx [target]
      in updateParameters net learningRate grads

-- Loss function (MSE)
meanSquaredError :: Vector Double -> Vector Double -> Double
meanSquaredError pred target =
  sum (map (^2) (zipWith (-) pred target)) / fromIntegral (length pred)

What to Expect

• Network should learn XOR within 1000–5000 epochs
• Loss should decrease (mostly) monotonically
• Verify by checking predictions on all four XOR inputs

Troubleshooting

Symptom	Likely cause	Fix
Loss doesn't decrease	Learning rate wrong	Try 0.1–1.0
Stuck at local minimum	Bad initialisation	Try different random seed
Numerical instability	Vanishing/exploding gradients	Check gradient norms