Learning Path: Understanding Model Behavior

Why do trained models behave as they do?

Explore the mechanisms and training influences behind the behavior we observe from finished models. This path examines knowledge, sampling, training and alignment, in-context adaptation, reasoning, and interpretability to connect what a model does at inference time with the processes that shaped that behavior.

Level: Intermediate → Advanced

Step 1 of 7

How Large Language Models (LLMs) Know Things They Were Never Taught

Web search, RAG, and the illusion of current knowledge
 •  Learning Paths:  How Large Language Models Work · Understanding Model Behavior  • Category:  AI and the Mathematics of Language  • Tags:  · · · ·
What You'll Learn
  • Why an LLM can answer questions about events that happened after its training ended
  • How web search gives a model current information without changing its weights
  • How RAG retrieves relevant information from private or specialized document collections
  • Why retrieved information is read during inference rather than learned by the model
  • How tool use lets an LLM work with search engines, code, databases, and external APIs
  • Why retrieval quality, source accuracy, and model reasoning all affect the final answer
Step 2 of 7

Temperature and Top-P: The Creativity Knobs

How sampling parameters shape AI personality
 •  Learning Paths:  How Large Language Models Work · Understanding Model Behavior  • Category:  AI and the Mathematics of Language  • Tags:  · · · · · · · · ·
What You'll Learn
  • How an LLM turns raw token scores into probabilities before choosing what comes next
  • How temperature reshapes a probability distribution to make outputs more predictable or more varied
  • How top-p sampling limits which tokens the model is allowed to consider
  • Why top-p adapts to model confidence differently from a fixed top-k cutoff
  • How temperature and top-p interact when both are applied to the same distribution
  • How to choose sampling settings for factual, structured, professional, and creative tasks
Step 3 of 7

Pre-training vs. Post-training

○ Planned

The distinction between the compute-heavy foundation training phase and the alignment phase that shapes model behavior (supervised fine-tuning, RLHF, DPO, Constitutional AI). One of the highest-confusion topics for readers new to the field and a natural companion to the cooperative witness piece.

Step 4 of 7

In-Context Learning

○ Planned

How models adapt to patterns within a single prompt without parameter updates, and why induction heads (covered in Inside Attention Part 1) are part but not all of the mechanistic story.

Step 5 of 7

The Cooperative Witness Problem

○ Planned

A piece on the ways language models tend to accept and continue user premises rather than push back on them, and the training dynamics that produce that tendency. Follows naturally from Part 2 of Inside Attention, which sets up the function-class framing this piece depends on.

Step 6 of 7

Reasoning Models and Test-Time Compute

○ Planned

The class of models that spend additional compute at inference time to improve their answers, and why this changes what 'capability' means.

Step 7 of 7

A Mechanistic Interpretability Primer

○ Planned

An introduction to the research program of reverse-engineering trained transformers, at the level of concrete circuits and features rather than high-level intuitions.