Setup Guide

Resources & Environment Setup

Everything you need before starting Module 1: tools, hardware, data, and verified links to all course materials.

Setup Checklist

  • ROCm installed and verified with rocm-smi
  • GHC 9.4+ and Cabal installed via ghcup
  • Rust installed via rustup
  • Shakespeare corpus downloaded (data/shakespeare.txt, ~5.4 MB)
  • Project structure created and cabal build succeeds
  • Rust GPU code compiles: cargo build --release
  • Can run a hello-world in GHCi
  • GPU seated and powered; sufficient VRAM (8GB+ recommended)
  • ~50GB free disk space

Hardware Requirements

ConfigurationMinimumRecommended
GPUAMD RDNA 2/3 (RX 6700+), 8GB VRAMRX 6800 XT or MI100, 16GB VRAM
System RAM8 GB16 GB+
Storage50 GB100 GB SSD
OSUbuntu 20.04+ / Fedora 34+Same
CPU-only training is possible but ~50–100× slower. The Shakespeare small model would take weeks per epoch instead of minutes.

Step 1: Install ROCm

Bash — Ubuntu / Debian
wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | sudo apt-key add -
echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/ubuntu focal main' \
  | sudo tee /etc/apt/sources.list.d/rocm.list

sudo apt update
sudo apt install -y rocm-dkms rocm-libs rocm-hipclang rocm-cmake

sudo usermod -a -G video $USER   # Log out and back in after this

# Verify
rocm-smi
hipcc --version
Bash — Fedora
sudo dnf install rocm-hip rocm-smi rocm-cmake
rocm-smi  # Verify

Step 2: Install Haskell (GHC + Cabal)

Bash
curl --proto '=https' --tlsv1.2 -sSf https://get-ghcup.haskell.org | sh

source ~/.ghcup/env  # Or restart your shell

ghc --version    # Recommended: 9.4.5+
cabal --version

Sample cabal project template

llm.cabal
cabal-version: 2.4
name:          llm-from-scratch
version:       0.1.0.0

library
  exposed-modules: Math, Layer, Transformer, Optimizer
  build-depends:   base >=4.16 && <5,
                   array, hmatrix, vector,
                   random, bytestring, binary
  hs-source-dirs:  src
  ghc-options:     -Wall -O2 -fllvm

executable llm-train
  main-is:       Main.hs
  build-depends: base >=4.16 && <5, llm-from-scratch, time
  hs-source-dirs: app
  ghc-options:   -Wall -O2 -threaded -rtsopts -with-rtsopts=-N

Step 3: Install Rust

Bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

rustc --version
cargo --version

# Create GPU kernel library
cargo new --lib llm-hip-kernels
Cargo.toml
[package]
name    = "llm-hip-kernels"
version = "0.1.0"
edition = "2021"

[lib]
crate-type = ["cdylib"]  # Shared library for Haskell FFI

Step 4: Download Shakespeare Corpus

Bash
mkdir -p data
wget https://raw.githubusercontent.com/brunoklein99/deep-learning-notes/master/shakespeare.txt \
  -O data/shakespeare.txt

# Verify (~5.4 MB)
wc -c data/shakespeare.txt
head -20 data/shakespeare.txt

Troubleshooting

ProblemLikely causeFix
rocm-smi shows no devices Driver issue sudo apt install --reinstall rocm-dkms && sudo reboot
GPU at 0% during training Using CPU fallback Verify hipcc --version works; check FFI linkage
Out of memory during training Batch too large Reduce microBatchSize; enable gradient checkpointing
Haskell stack overflow at compile GHC default stack export HSTACKSIZE=256m && cabal build
Missing Haskell dependencies Stale index cabal update && cabal clean && cabal build

Complete Paper List

Foundational (read in order)

Learning Representations by Back-Propagating Errors (1986) Rumelhart, Hinton, Williams — Nature 323 doi.org/10.1038/323533a0
Long Short-Term Memory (1997) Hochreiter & Schmidhuber — Neural Computation 9(8) MIT Press
Attention Is All You Need (2017) Vaswani et al. — NeurIPS arxiv.org/abs/1706.03762
Language Models are Unsupervised Multitask Learners (GPT-2, 2019) Radford, Wu et al. PDF
Language Models are Few-Shot Learners (GPT-3, 2020) Brown et al. arxiv.org/abs/2005.14165
Training Compute-Optimal Large Language Models (Chinchilla, 2022) Hoffmann et al. arxiv.org/abs/2203.15556

Supplementary papers

Adam: A Method for Stochastic Optimisation (2014) Kingma & Ba arxiv.org/abs/1412.6980
An Overview of Gradient Descent Optimisation Algorithms (2016) Sebastian Ruder arxiv.org/abs/1609.04747
On the Difficulty of Training RNNs (2001) Hochreiter, Bengio et al. arxiv.org/abs/1211.1541
Neural Machine Translation by Jointly Learning to Align and Translate (2014) Bahdanau, Cho, Bengio arxiv.org/abs/1409.0473
Scaling Laws for Neural Language Models (2020) Kaplan et al. arxiv.org/abs/2001.08361
Root Mean Square Layer Normalisation (2019) Zhang & Sennrich arxiv.org/abs/1910.07468
GLU Variants Improve Transformer (2020) Noam Shazeer arxiv.org/abs/2002.05202
RoFormer: Enhanced Transformer with Rotary Position Embedding (2021) Su, Lu et al. arxiv.org/abs/2104.09864
Longformer: The Long-Document Transformer (2020) Beltagy, Peters, Cohan arxiv.org/abs/2004.05150
LLM.int8(): 8-bit Matrix Multiplication for Transformers (2022) Dettmers et al. arxiv.org/abs/2208.07339
Training Language Models to Follow Instructions (InstructGPT, 2022) Ouyang et al. arxiv.org/abs/2203.02155
Automatic Differentiation in Machine Learning: A Survey (2018) Baydin et al. arxiv.org/abs/1502.05767

Books

Introduction to Linear Algebra (5th ed.) Gilbert Strang math.mit.edu/~gs/linearalgebra/
Neural Networks and Deep Learning (free online) Michael Nielsen neuralnetworksanddeeplearning.com
Deep Learning Goodfellow, Bengio, Courville — MIT Press, free online deeplearningbook.org
Mathematics for Machine Learning (free PDF) Deisenroth, Faisal, Ong mml-book.github.io

Video series

MIT 18.06 Linear Algebra Gilbert Strang YouTube playlist
3Blue1Brown — Neural Networks YouTube playlist
Stanford CS231n — CNNs for Visual Recognition Andrej Karpathy et al. cs231n.stanford.edu
Stanford CS224n — NLP with Deep Learning Christopher Manning et al. web.stanford.edu/class/cs224n/
Yannic Kilcher — Paper Explanations Particularly: "Attention Is All You Need" and BERT explainers. YouTube channel

Blogs & interactive

The Illustrated Transformer Jay Alammar jalammar.github.io/illustrated-transformer/
Chris Olah's Blog Understanding LSTMs, Backpropagation, Functional Programming & NNs. colah.github.io
Lil'Log (Lilian Weng) Attention, policy gradients, meta-learning — deep technical content. lilianweng.github.io
Distill.pub Interactive ML explanations with excellent visualisations. distill.pub
The Annotated Transformer Harvard NLP nlp.seas.harvard.edu
nanoGPT Andrej Karpathy — reference implementation github.com/karpathy/nanoGPT