0 of 53 lessons completed
Home / Build an LLM from Scratch in PyTorch
Build an LLM from Scratch in PyTorch
Build a small GPT by hand: character tokens, embeddings with position, self-attention and transformer blocks, a training loop, then generation, sampling and a tiny BPE — ending in a trained capstone. Builds on PyTorch Intermediate.
Start learning ❯❯53 lessons · 15 topics · 0/53 completed
Course contents
Work top to bottom — the sidebar is the course. Topics later in the list assume the ones before them.
PyTorch The Road Ahead
Intermediate · ⏱ 8 min · standalone lesson
Text as Data
- PyTorch The Tiny Corpus
- PyTorch Characters and Vocabulary
- PyTorch Encode and Decode
- PyTorch Context and Target
- PyTorch Train and Validation Text
The Bigram Baseline
Start: PyTorch The Bigram by Counting ❯Averaging Context
Start: PyTorch The Bigram Ceiling ❯Embeddings with Position
- PyTorch Token Embeddings
- PyTorch Positional Embeddings
- PyTorch Adding Token and Position
- PyTorch Embedding Size Choices
Self-Attention
- PyTorch Tokens That Talk
- PyTorch Queries, Keys and Values
- PyTorch Scaled Dot-Product Attention
- PyTorch The Causal Mask
- PyTorch A Single Head
- PyTorch Reading the Weights
Multi-Head Attention
Start: PyTorch Many Heads in Parallel ❯The Transformer Block
- PyTorch The Feed-Forward Part
- PyTorch Residual Connections
- PyTorch LayerNorm by Hand
- PyTorch The Block Class
Assembling GPT
- PyTorch The GPT Class
- PyTorch Stacking Blocks
- PyTorch Counting Parameters
- PyTorch Logits to Loss
- PyTorch The First Loss Check
Training the Model
- PyTorch Getting Batches
- PyTorch The LLM Training Loop
- PyTorch Watching Two Losses
- PyTorch Overfitting the Corpus
- PyTorch Saving the Model
Generating Text
- PyTorch Greedy Generation
- PyTorch Sampling with Temperature
- PyTorch Top-k Sampling
- PyTorch Prompting the Model
- PyTorch Reading the Output Honestly
Better Tokens
- PyTorch Why Characters Are Slow
- PyTorch Counting Pairs
- PyTorch Learning Merges
- PyTorch Encoding with BPE
Making It Better
Start: PyTorch A Bigger Context ❯PyTorch Capstone: The Final Model
Advanced · ⏱ 15 min · standalone lesson
PyTorch What to Learn Next
Advanced · ⏱ 8 min · standalone lesson