EPOCH v0 - Kickoff Meeting

---

The Course

Stanford CS336: Language Modelling From Scratch

Topics covered:

1. Transformer architectures and their implementations in PyTorch
2. Distributed training
3. GPU kernels for optimizing inference algorithms
4. Data wrangling techniques for LLM training
5. RL algorithms for post-training alignment and reasoning

---

Prerequisites

Fair Warning: This is a hard course

The website lists prerequisites that may even be understating the difficulty

1. Proficiency in Python
2. Experience with deep learning and systems optimization
3. College Calculus and Linear Algebra
4. Basic Probability and Statistics
5. Foundational Machine Learning knowledge

---

Prerequisites (cont.)

Learning under pressure leads to the best outcomes.

Resources available:

• Documentation
• Papers
• Communities
• Each other

---

Course Structure

5 Assignments Total

Meetings every Wednesday to discuss the previous assignment.

---

Accountability

A public list of assignment completions will be maintained and published on the website.

---

Compute Requirements

Some assignments require paid compute (GPU access) for full implementation.

GPU: Ideal, but costs money

CPU: Possible for some work if you have enough memory — slower, but workable for certain tasks.

---

Assignment 1

1. Follow lectures through Lecture 5
2. Implement the core components: tokenizer, model architecture, and optimizer
3. Train a minimal language model

See you on 10th December!