Inside Attention, Part 1: The Mechanism

Attention is the engine. The rest of the transformer architecture stabilizes it, organizes it, and makes deep training possible.
 • Category: AI and the Mathematics of Language,  • Tags: #AI, #attention mechanisms, #induction heads, #interpretability, #LLM, #softmax, #transformers
What You'll Learn
  • How queries, keys, and values work together to let tokens exchange information
  • Why attention scores are divided by the square root of the key dimension
  • How scaling prevents softmax from saturating and preserves useful gradient flow
  • Why transformers split attention across multiple heads instead of using one large attention operation
  • What researchers have discovered about the specialized roles learned by attention heads
  • How induction heads learn a match-and-copy algorithm that helps explain in-context learning

The Discrete Mathematics Hiding Inside LLMs

How set theory, predicate logic, and formal proofs show up in modern AI
 • Category: AI and the Mathematics of Language,  • Tags: #AI, #attention mechanisms, #embeddings, #discrete mathematics, #LLM, #predicate logic, #set theory
What You'll Learn
  • How attention behaves like a soft version of predicate logic
  • How top-k and top-p sampling use ideas from set theory
  • How Boolean logic determines which tokens can attend to each other
  • Why chain-of-thought reasoning resembles the structure of a proof
  • How positional encoding uses periodic patterns to represent position
  • How discrete math helps explain why common LLM techniques work

How Large Language Models (LLMs) Know Things They Were Never Taught

Web search, RAG, and the illusion of current knowledge
 • Category: AI and the Mathematics of Language,  • Tags: #AI, #LLM, #RAG, #web search, #context windows
What You'll Learn
  • Why an LLM can answer questions about events that happened after its training ended
  • How web search gives a model current information without changing its weights
  • How RAG retrieves relevant information from private or specialized document collections
  • Why retrieved information is read during inference rather than learned by the model
  • How tool use lets an LLM work with search engines, code, databases, and external APIs
  • Why retrieval quality, source accuracy, and model reasoning all affect the final answer

Temperature and Top-P: The Creativity Knobs

How sampling parameters shape AI personality
 • Category: AI and the Mathematics of Language,  • Tags: #AI, #Anthropic, #LLM, #nucleus sampling, #OpenAI, #probability, #sampling, #softmax, #temperature, #top-p
What You'll Learn
  • How an LLM turns raw token scores into probabilities before choosing what comes next
  • How temperature reshapes a probability distribution to make outputs more predictable or more varied
  • How top-p sampling limits which tokens the model is allowed to consider
  • Why top-p adapts to model confidence differently from a fixed top-k cutoff
  • How temperature and top-p interact when both are applied to the same distribution
  • How to choose sampling settings for factual, structured, professional, and creative tasks