Learning Path: Inside the Transformer

What's actually happening inside the architecture?

Look beneath the high-level description of a transformer and examine the machinery that makes it work. This path moves from vector representations into attention, feedforward networks, and other internal mechanisms, building toward the ability to understand—and eventually implement—the architecture rather than treating it as a black box.

Level: Foundational → Advanced

Step 1 of 6

How Large Language Models (LLMs) Think: Turning Meaning into Math

Exploring how large language models use linear algebra to create geometric meaning
 •  Learning Paths:  How Large Language Models Work · Inside the Transformer  • Category:  AI and the Mathematics of Language  • Tags:  · · · ·
What You'll Learn
  • How an LLM turns words and tokens into numerical vectors it can process
  • How distance and direction in embedding space can represent relationships in meaning
  • Why linear algebra and geometry are two ways of describing the same internal structure
  • How matrix operations transform information as it moves through a model
  • Why high-dimensional spaces can represent many subtle features of language at once
  • How probability guides a model from its current context toward the next token
Step 2 of 6

The Discrete Mathematics Hiding Inside LLMs

How set theory, predicate logic, and formal proofs show up in modern AI
 •  Learning Paths:  How Large Language Models Work · Inside the Transformer  • Category:  AI and the Mathematics of Language  • Tags:  · · · · · ·
What You'll Learn
  • How attention behaves like a soft version of predicate logic
  • How top-k and top-p sampling use ideas from set theory
  • How Boolean logic determines which tokens can attend to each other
  • Why chain-of-thought reasoning resembles the structure of a proof
  • How positional encoding uses periodic patterns to represent position
  • How discrete math helps explain why common LLM techniques work
Step 3 of 6

Inside Attention, Part 1: The Mechanism

Attention is the engine. The rest of the transformer architecture stabilizes it, organizes it, and makes deep training possible.
 •  Learning Paths:  How Large Language Models Work · Inside the Transformer  • Category:  AI and the Mathematics of Language  • Tags:  · · · · · ·
What You'll Learn
  • How queries, keys, and values work together to let tokens exchange information
  • Why attention scores are divided by the square root of the key dimension
  • How scaling prevents softmax from saturating and preserves useful gradient flow
  • Why transformers split attention across multiple heads instead of using one large attention operation
  • What researchers have discovered about the specialized roles learned by attention heads
  • How induction heads learn a match-and-copy algorithm that helps explain in-context learning
Step 4 of 6

The Feedforward Sublayer

○ Planned

A standalone post on the FFN sublayer inside each transformer block: what it does, why it is often much larger than the attention sublayer, and what interpretability research has shown about the concepts stored there.

Step 5 of 6

A Mechanistic Interpretability Primer

○ Planned

An introduction to the research program of reverse-engineering trained transformers, at the level of concrete circuits and features rather than high-level intuitions.

Step 6 of 6 · Capstone

Implementing a Minimal Transformer in PyTorch

◐ In Progress

Building the core machinery of a language model from embeddings and attention to training and generation