PyTorch What to Learn Next

A learning map is a list of the next skills to study after a course ends. This map points to real tokenizers, the library transformer in PyTorch Advanced, and a BPE retrain project.

A short list keeps its order

A list keeps the order its items were added in. The sample below holds the first four lesson names of this course in order.

slugs = ["llm-intro", "the-tiny-corpus", "characters-and-vocab", "encode-and-decode"]
print(len(slugs))
print(slugs[0])
print(slugs[-1])

Checklist lines carry the course numbers

Numbering a checklist line with its course position keeps the order visible. The sample below prints lessons thirty to thirty-two in checklist form.

slugs = ["the-gpt-class", "stacking-blocks", "counting-parameters"]
for number, slug in enumerate(slugs, start=30):
    print(f"{number:02d}. [ ] {slug}")

Finding one lesson in the list

An index lookup reports the position of one item in a list. In the sample below, the BPE encoding lesson sits at index one.

slugs = ["learning-merges", "encoding-with-bpe", "a-bigger-context", "dropout-in-the-model"]
print(slugs.index("encoding-with-bpe"))
print("a-bigger-context" in slugs)

Example

slugs = [
    "llm-intro",
    "the-tiny-corpus",
    "characters-and-vocab",
    "encode-and-decode",
    "context-and-target",
    "train-and-validation-text",
    "bigram-by-counting",
    "bigram-as-an-embedding",
    "training-the-bigram",
    "the-bigram-ceiling",
    "averaging-previous-tokens",
    "weighted-averaging",
    "token-embeddings",
    "positional-embeddings",
    "adding-token-and-position",
    "embedding-size-choices",
    "tokens-that-talk",
    "queries-keys-and-values",
    "scaled-dot-product",
    "the-causal-mask",
    "a-single-head",
    "reading-the-weights",
    "many-heads-in-parallel",
    "the-multi-head-module",
    "heads-and-sizes",
    "the-feed-forward-part",
    "residual-connections",
    "layernorm-by-hand",
    "the-block-class",
    "the-gpt-class",
    "stacking-blocks",
    "counting-parameters",
    "logits-to-loss",
    "the-first-loss-check",
    "getting-batches",
    "the-llm-training-loop",
    "watching-two-losses",
    "overfitting-the-corpus",
    "saving-the-model",
    "greedy-generation",
    "sampling-with-temperature",
    "top-k-sampling",
    "prompting-the-model",
    "reading-the-output-honestly",
    "why-characters-are-slow",
    "counting-pairs",
    "learning-merges",
    "encoding-with-bpe",
    "a-bigger-context",
    "dropout-in-the-model",
    "what-scale-changes",
    "llm-capstone",
    "what-to-learn-next",
]

print("Course checklist")
for number, slug in enumerate(slugs, start=1):
    print(f"{number:02d}. [ ] {slug}")
print(f"Total lessons: {len(slugs)}")

Key points

Tokenizer: a program that splits text into tokens and numbers them
Real tokenizers split text into subwords, not single characters
BPE: the merge method that builds subword tokens from frequent pairs
A BPE retrain swaps character tokens for merged tokens in this same GPT
PyTorch Advanced covers the transformer from the library side
The checklist below names every lesson of this course once, in order

In this example, the program embeds the slug list of this whole course in course order. It prints each slug once as a numbered checklist line with an empty box. It prints the total at the end, and the total is 53. The list runs from the first lesson of the course to this lesson itself. Working through the unchecked boxes in order is the revision path for the course.

Note

The retrain project reuses the capstone model code with a BPE token set

Try it Yourself

Edit the code, then press Run.

Running is simulated · Python Playground

slugs = [
    "llm-intro",
    "the-tiny-corpus",
    "characters-and-vocab",
    "encode-and-decode",
    "context-and-target",
    "train-and-validation-text",
    "bigram-by-counting",
    "bigram-as-an-embedding",
    "training-the-bigram",
    "the-bigram-ceiling",
    "averaging-previous-tokens",
    "weighted-averaging",
    "token-embeddings",
    "positional-embeddings",
    "adding-token-and-position",
    "embedding-size-choices",
    "tokens-that-talk",
    "queries-keys-and-values",
    "scaled-dot-product",
    "the-causal-mask",
    "a-single-head",
    "reading-the-weights",
    "many-heads-in-parallel",
    "the-multi-head-module",
    "heads-and-sizes",
    "the-feed-forward-part",
    "residual-connections",
    "layernorm-by-hand",
    "the-block-class",
    "the-gpt-class",
    "stacking-blocks",
    "counting-parameters",
    "logits-to-loss",
    "the-first-loss-check",
    "getting-batches",
    "the-llm-training-loop",
    "watching-two-losses",
    "overfitting-the-corpus",
    "saving-the-model",
    "greedy-generation",
    "sampling-with-temperature",
    "top-k-sampling",
    "prompting-the-model",
    "reading-the-output-honestly",
    "why-characters-are-slow",
    "counting-pairs",
    "learning-merges",
    "encoding-with-bpe",
    "a-bigger-context",
    "dropout-in-the-model",
    "what-scale-changes",
    "llm-capstone",
    "what-to-learn-next",
]

print("Course checklist")
for number, slug in enumerate(slugs, start=1):
    print(f"{number:02d}. [ ] {slug}")
print(f"Total lessons: {len(slugs)}")

Exercise

What does the BPE retrain project change in the capstone model?

The token set The loss function The optimizer The number of blocks