Mathematical Foundations

1.1 Linear Algebra Essentials

Key Concepts

• Vectors, matrices, and tensors
• Matrix multiplication and broadcasting
• Eigenvalues, eigenvectors, and diagonalisation
• Singular Value Decomposition (SVD)

Implementation Task 1.1 — Matrix Library

Build a matrix library in Haskell with: basic operations (add, multiply, transpose), broadcasting rules, and element-wise operations.

module Matrix where

newtype Matrix a = Matrix [[a]]  -- Simple representation to start

-- For production, use more efficient arrays
import Data.Array.Unboxed (UArray, array, bounds, (!))

data DenseMatrix = DenseMatrix
  { rows  :: Int
  , cols  :: Int
  , data_ :: UArray (Int, Int) Double
  }

matmul :: DenseMatrix -> DenseMatrix -> DenseMatrix
matmul m1 m2 = -- Implementation using dot products

1.2 Calculus for Machine Learning

Key Concepts

• Partial derivatives and gradients
• Chain rule (critical for backpropagation)
• Optimisation and critical points
• Convexity and local minima

Implementation Task 1.2 — Automatic Differentiation

{-# LANGUAGE GeneralizedNewtypeDeriving #-}

-- Simple forward-mode AD
data Dual a = Dual { primal :: a, tangent :: a }
  deriving (Show, Eq)

instance Num a => Num (Dual a) where
  Dual p1 t1 + Dual p2 t2 = Dual (p1 + p2) (t1 + t2)
  Dual p1 t1 * Dual p2 t2 = Dual (p1 * p2) (p1 * t2 + t1 * p2)
  -- ... etc

-- Reverse-mode AD (more useful for neural networks) is more complex.
-- Consider implementing a tape-based system.

1.3 Gradient Descent and Optimisation

Key Concepts

• Batch gradient descent, SGD, mini-batch GD
• Learning rates and scheduling
• Momentum and Nesterov acceleration
• Adam optimiser and adaptive learning rates

Implementation Task 1.3 — Multiple Optimisers

data OptimizerState = SGDState
  { learningRate :: Double
  , momentum     :: Double
  , velocity     :: [Double]  -- Or Matrix for full network
  }

-- Update parameters given gradients
updateSGD :: OptimizerState -> [Double] -> [Double] -> (OptimizerState, [Double])
updateSGD state params grads =
  let newVel    = zipWith (\v g -> momentum state * v - learningRate state * g)
                          (velocity state) grads
      newParams = zipWith (+) params newVel
  in (state { velocity = newVel }, newParams)

Adam formulas