Building an LLM from Scratch

A practical, implementation-focused course covering everything from matrix operations to training a 500M-parameter language model on Shakespeare — in Haskell and Rust.

Duration: 3–4 months Pace: 20–30 hrs / week Hardware: AMD GPU (ROCm) Languages: Haskell + Rust

About This Course

This course is designed for experienced programmers who want to understand large language models through implementation, not just theory. By building working examples in Haskell — with Rust for performance-critical GPU/ROCm code — you develop deep understanding of what's actually happening inside modern AI systems.

You'll progress from basic linear algebra to training a real language model capable of generating Shakespeare-like text. Every concept is grounded in code you write yourself.

What	Details
Prerequisites	programming experience, Linux/systems comfort, mathematical maturity (calculus + linear algebra, OK if rusty). No prior ML knowledge required.
Final project	Train a 500M–1B parameter language model on Shakespeare's complete works using AMD GPU + ROCm
Hardware	AMD RDNA 2/3 GPU, 8GB+ VRAM recommended. CPU-only training is possible but ~100× slower.
Languages	Haskell (training logic, neural network code) + Rust (GPU kernels via HIP/ROCm)

Course Structure

Twelve modules in four logical phases, each building on the last:

Module 1

Mathematical Foundations

Linear algebra, calculus, gradient descent. Build a matrix library in Haskell.

2 weeks · Deliverable: Matrix library

Module 2

Neural Networks & Backprop

Perceptron, MLP, backpropagation. Train a network to solve XOR.

2 weeks · Deliverable: MLP solving XOR

Module 3

CNNs & Recurrent Networks

Convolutional networks, RNNs, the vanishing gradient problem.

2 weeks · Context-setting module

Module 4

LSTM

Gating mechanisms, LSTM cells, character-level language modelling.

2 weeks · Deliverable: LSTM char-level LM

Module 5

Attention Mechanisms

Scaled dot-product attention, multi-head attention, masked attention.

1–2 weeks · Deliverable: Attention layer

Module 6

Transformer Architecture

Positional encoding, transformer blocks, layer norm, feed-forward networks.

2 weeks · Deliverable: Full transformer block

Module 7

Training at Scale

Next-token prediction, training loops, LR schedules, checkpointing.

1 week · Deliverable: Training loop

Module 8

Small Language Model

Build and train a character-level language model (~40M params) on Shakespeare.

2 weeks · Deliverable: Generating text

Module 9

Scaling to Larger Models

Chinchilla scaling laws, mixed precision, gradient accumulation, multi-GPU concepts.

1–2 weeks

Module 10

AMD ROCm & GPU Training

ROCm setup, HIP programming, matrix ops in Rust, Haskell–Rust FFI bridge.

2 weeks · Deliverable: GPU kernels

Module 11

Training on Shakespeare

Full 500M–1B parameter model. Distributed training, refinement, evaluation.

4 weeks · Final project

Module 12

Advanced Topics

RMSNorm, SwiGLU, RoPE, sparse attention, quantisation. Ongoing exploration.

Ongoing

Learning Path

The modules are designed to be followed sequentially — concepts compound. Rough phase breakdown:

Phase	Modules	Time	Focus
Foundations	1–2	Weeks 1–4	Maths, neural networks, backprop
Sequence models	3–4	Weeks 5–8	RNNs, LSTMs, gating
Transformers	5–7	Weeks 9–12	Attention, architecture, training
Real models	8–11	Weeks 13–20	Scaling, GPU, full Shakespeare training
Extensions	12	Weeks 21+	Advanced techniques, research reading

Download

The original course documents are available as a single archive: course.zip

Before You Begin

Set up your environment before starting Module 1. The Resources & Setup page has step-by-step instructions for installing ROCm, GHC, Cabal, and Rust — plus download links for the Shakespeare corpus and all verified paper/video references.

Start with Resources & Setup, then Module 1. Each module page lists its reading and viewing recommendations.