Learning Path: How Large Language Models Work

How does an LLM work from input to output?

Follow the major stages of an LLM from the moment text enters the system to the generation of a response. This path builds from tokenization and mathematical representations through learning, attention, context, and inference, with each article adding another piece to an end-to-end mental model of how an LLM works.

Level: Foundational → Capstone

Step 1 of 14

How Large Language Models (LLMs) Tokenize Text: Why Words Aren't What You Think

Understanding how LLMs break language into pieces—and why it matters more than you realize
 •  Learning Path:  How Large Language Models Work  • Category:  AI and the Mathematics of Language  • Tags:  · · · ·
What You'll Learn
  • Why language models break text into tokens instead of reading whole words
  • How subword tokenization balances vocabulary size with the amount of text a model must process
  • How Byte Pair Encoding (BPE) learns useful token boundaries from patterns in training data
  • Why the same sentence can use very different numbers of tokens across languages, code, and rare words
  • How tokenization can cause surprising failures in spelling, letter counting, and unusual inputs
  • Why token counts affect context limits, processing efficiency, and the cost of using an LLM
Step 2 of 14

How Large Language Models (LLMs) Think: Turning Meaning into Math

Exploring how large language models use linear algebra to create geometric meaning
 •  Learning Paths:  How Large Language Models Work · Inside the Transformer  • Category:  AI and the Mathematics of Language  • Tags:  · · · ·
What You'll Learn
  • How an LLM turns words and tokens into numerical vectors it can process
  • How distance and direction in embedding space can represent relationships in meaning
  • Why linear algebra and geometry are two ways of describing the same internal structure
  • How matrix operations transform information as it moves through a model
  • Why high-dimensional spaces can represent many subtle features of language at once
  • How probability guides a model from its current context toward the next token
Step 3 of 14

The Discrete Mathematics Hiding Inside LLMs

How set theory, predicate logic, and formal proofs show up in modern AI
 •  Learning Paths:  How Large Language Models Work · Inside the Transformer  • Category:  AI and the Mathematics of Language  • Tags:  · · · · · ·
What You'll Learn
  • How attention behaves like a soft version of predicate logic
  • How top-k and top-p sampling use ideas from set theory
  • How Boolean logic determines which tokens can attend to each other
  • Why chain-of-thought reasoning resembles the structure of a proof
  • How positional encoding uses periodic patterns to represent position
  • How discrete math helps explain why common LLM techniques work
Step 4 of 14

How Large Language Models (LLMs) Learn: Calculus and the Search for Understanding

Exploring how gradient descent and partial derivatives teach models to think
 •  Learning Paths:  How Large Language Models Work · How LLMs Learn and Acquire Capabilities  • Category:  AI and the Mathematics of Language  • Tags:  · · · ·
What You'll Learn
  • How derivatives tell a model which direction will reduce its prediction error
  • How gradient descent turns billions of small corrections into learning
  • How backpropagation uses the chain rule to assign error across many layers
  • Why the learning rate controls the balance between fast progress and stable training
  • Why noisy mini-batch updates can help a model generalize instead of memorize
  • How transformers keep gradients stable while learning which patterns deserve attention
Step 5 of 14

Softmax and Cross-Entropy Loss

○ Planned

A focused post on the connective tissue between probability and learning. Softmax as the operation that turns real numbers into distributions; cross-entropy as the loss that measures how well those distributions match the truth. Short, foundational, unlocks the probability layer.

Step 6 of 14

Inside Attention, Part 1: The Mechanism

Attention is the engine. The rest of the transformer architecture stabilizes it, organizes it, and makes deep training possible.
 •  Learning Paths:  How Large Language Models Work · Inside the Transformer  • Category:  AI and the Mathematics of Language  • Tags:  · · · · · ·
What You'll Learn
  • How queries, keys, and values work together to let tokens exchange information
  • Why attention scores are divided by the square root of the key dimension
  • How scaling prevents softmax from saturating and preserves useful gradient flow
  • Why transformers split attention across multiple heads instead of using one large attention operation
  • What researchers have discovered about the specialized roles learned by attention heads
  • How induction heads learn a match-and-copy algorithm that helps explain in-context learning
Step 7 of 14

The Feedforward Sublayer

○ Planned

A standalone post on the FFN sublayer inside each transformer block: what it does, why it is often much larger than the attention sublayer, and what interpretability research has shown about the concepts stored there.

Step 8 of 14

How Large Language Models (LLMs) Handle Context Windows: The Memory That Isn't Memory

Exploring why longer context doesn't mean better memory and what happens when conversations grow
 •  Learning Paths:  AI for Software Engineers · How Large Language Models Work  • Category:  AI and the Mathematics of Language  • Tags:  · · · ·
What You'll Learn
  • Why an LLM’s context window is not the same thing as memory
  • How chat applications create continuity even though the underlying model is stateless
  • How attention lets earlier parts of a conversation influence the next token
  • Why longer conversations become increasingly expensive for a transformer to process
  • Why information can become harder to use even while it remains inside the context window
  • How truncation, summarization, retrieval, and KV caching help manage long conversations
Step 9 of 14

Context Compression: What Does 'Lossless' Really Mean?

○ Planned

As context windows fill, AI systems increasingly rely on summarization and compression to preserve what matters. But if a system must decide what to discard before it knows what will matter later, how “lossless” can that compression really be? An examination of what context compression preserves, what it inevitably risks losing, and why the distinction matters for long-running AI systems.

Step 10 of 14

In-Context Learning

○ Planned

How models adapt to patterns within a single prompt without parameter updates, and why induction heads (covered in Inside Attention Part 1) are part but not all of the mechanistic story.

Step 11 of 14

How Large Language Models (LLMs) Know Things They Were Never Taught

Web search, RAG, and the illusion of current knowledge
 •  Learning Paths:  How Large Language Models Work · Understanding Model Behavior  • Category:  AI and the Mathematics of Language  • Tags:  · · · ·
What You'll Learn
  • Why an LLM can answer questions about events that happened after its training ended
  • How web search gives a model current information without changing its weights
  • How RAG retrieves relevant information from private or specialized document collections
  • Why retrieved information is read during inference rather than learned by the model
  • How tool use lets an LLM work with search engines, code, databases, and external APIs
  • Why retrieval quality, source accuracy, and model reasoning all affect the final answer
Step 12 of 14

Temperature and Top-P: The Creativity Knobs

How sampling parameters shape AI personality
 •  Learning Paths:  How Large Language Models Work · Understanding Model Behavior  • Category:  AI and the Mathematics of Language  • Tags:  · · · · · · · · ·
What You'll Learn
  • How an LLM turns raw token scores into probabilities before choosing what comes next
  • How temperature reshapes a probability distribution to make outputs more predictable or more varied
  • How top-p sampling limits which tokens the model is allowed to consider
  • Why top-p adapts to model confidence differently from a fixed top-k cutoff
  • How temperature and top-p interact when both are applied to the same distribution
  • How to choose sampling settings for factual, structured, professional, and creative tasks
Step 13 of 14

Scaling Laws and Emergence

○ Planned

The empirical relationships between compute, data, parameters, and capability, and where the sharp transitions in behavior come from.

Step 14 of 14 · Capstone

Implementing a Minimal Transformer in PyTorch

◐ In Progress

Building the core machinery of a language model from embeddings and attention to training and generation