What You'll Learn
- How queries, keys, and values work together to let tokens exchange information
- Why attention scores are divided by the square root of the key dimension
- How scaling prevents softmax from saturating and preserves useful gradient flow
- Why transformers split attention across multiple heads instead of using one large attention operation
- What researchers have discovered about the specialized roles learned by attention heads
- How induction heads learn a match-and-copy algorithm that helps explain in-context learning
What You'll Learn
- Why language models break text into tokens instead of reading whole words
- How subword tokenization balances vocabulary size with the amount of text a model must process
- How Byte Pair Encoding (BPE) learns useful token boundaries from patterns in training data
- Why the same sentence can use very different numbers of tokens across languages, code, and rare words
- How tokenization can cause surprising failures in spelling, letter counting, and unusual inputs
- Why token counts affect context limits, processing efficiency, and the cost of using an LLM
What You'll Learn
- Why an LLM’s context window is not the same thing as memory
- How chat applications create continuity even though the underlying model is stateless
- How attention lets earlier parts of a conversation influence the next token
- Why longer conversations become increasingly expensive for a transformer to process
- Why information can become harder to use even while it remains inside the context window
- How truncation, summarization, retrieval, and KV caching help manage long conversations
![]()
