2 "Context Windows" Posts

How Large Language Models (LLMs) Know Things They Were Never Taught

Web search, RAG, and the illusion of current knowledge

Feb 9, 2026 • Learning Paths: How Large Language Models Work · Understanding Model Behavior • Category: AI and the Mathematics of Language • Tags: #AI · #LLM · #RAG · #web search · #context windows

What You'll Learn

Why an LLM can answer questions about events that happened after its training ended
How web search gives a model current information without changing its weights
How RAG retrieves relevant information from private or specialized document collections
Why retrieved information is read during inference rather than learned by the model
How tool use lets an LLM work with search engines, code, databases, and external APIs
Why retrieval quality, source accuracy, and model reasoning all affect the final answer

How Large Language Models (LLMs) Handle Context Windows: The Memory That Isn't Memory

Exploring why longer context doesn't mean better memory and what happens when conversations grow

Nov 10, 2025 • Learning Paths: AI for Software Engineers · How Large Language Models Work • Category: AI and the Mathematics of Language • Tags: #AI · #attention mechanisms · #context windows · #LLM · #transformers

What You'll Learn

Why an LLM’s context window is not the same thing as memory
How chat applications create continuity even though the underlying model is stateless
How attention lets earlier parts of a conversation influence the next token
Why longer conversations become increasingly expensive for a transformer to process
Why information can become harder to use even while it remains inside the context window
How truncation, summarization, retrieval, and KV caching help manage long conversations

Behind the Code

My focus is the engineering beneath modern AI systems. After several years of building applications with AI through APIs, my interests have moved below that abstraction layer: understanding how models are architected, trained, evaluated, and served, why they’re designed the way they are, and the engineering tradeoffs behind those decisions.

That direction follows a pattern across my forty-year career as a software developer. I’ve primarily built applications, but I’ve repeatedly found myself pushing beyond the application itself and building the systems, abstractions, and tooling behind it: metaprogramming layers, dynamic runtimes, configuration-driven engines, and automation platforms designed to outlive the original feature request.

The pattern shows across my career: Forms Express at AT&T, a four-layer dynamic runtime shipped for under $500K against a $2M consulting bid; VeriSign’s first-ever customer self-service mobile activation system, which won the 2003 SUPERQuest Award; and TAG at Microsoft, a production LLM orchestration platform that shipped nearly 100 Terraform articles by parsing XML templates, routing prompts to Azure OpenAI, and composing results with browser-driven Azure portal capture. Along the way: C++ MVP, nine technology books, 100+ articles for .NET Programming Tips & Techniques, and helping run CodeGuru, the largest Windows developer community of its era.

After 21 years at Microsoft, I’m now full-time on what’s next: completing my Master’s in Computer Science and publishing at Signal & Syntax, where I research and write about the mathematics and engineering underneath modern AI. I use mathematical models, experiments, and small implementations to explore everything from gradient descent and attention to inference-time behavior and training infrastructure; not just how to use these systems through an API, but how the machinery beneath that API actually works.

Connect with me on LinkedIn .