2 "Tokenization" Posts

How Large Language Models (LLMs) Tokenize Text: Why Words Aren't What You Think

Understanding how LLMs break language into pieces—and why it matters more than you realize

Nov 11, 2025 • Learning Path: How Large Language Models Work • Category: AI and the Mathematics of Language • Tags: #AI · #tokenization · #LLM · #transformers · #NLP

What You'll Learn

Why language models break text into tokens instead of reading whole words
How subword tokenization balances vocabulary size with the amount of text a model must process
How Byte Pair Encoding (BPE) learns useful token boundaries from patterns in training data
Why the same sentence can use very different numbers of tokens across languages, code, and rare words
How tokenization can cause surprising failures in spelling, letter counting, and unusual inputs
Why token counts affect context limits, processing efficiency, and the cost of using an LLM

How Large Language Models (LLMs) Read Code: Seeing Patterns Instead of Logic

Exploring how large language models interpret code and what they miss

Oct 6, 2025 • Learning Path: AI for Software Engineers • Category: AI and the Mathematics of Language • Tags: #AI · #LLM · #probability · #tokenization

What You'll Learn

How an LLM reads code differently from a compiler or a human developer
Why models recognize programming patterns instead of executing the code they see
How embeddings let an LLM associate code with similar structures and meanings
Why comments, variable names, and familiar coding idioms can change a model’s interpretation
How statistically likely code can still be logically or operationally wrong
Why combining generative AI with compilers and static analysis produces safer coding tools

Behind the Code

My focus is the engineering beneath modern AI systems. After several years of building applications with AI through APIs, my interests have moved below that abstraction layer: understanding how models are architected, trained, evaluated, and served, why they’re designed the way they are, and the engineering tradeoffs behind those decisions.

That direction follows a pattern across my forty-year career as a software developer. I’ve primarily built applications, but I’ve repeatedly found myself pushing beyond the application itself and building the systems, abstractions, and tooling behind it: metaprogramming layers, dynamic runtimes, configuration-driven engines, and automation platforms designed to outlive the original feature request.

The pattern shows across my career: Forms Express at AT&T, a four-layer dynamic runtime shipped for under $500K against a $2M consulting bid; VeriSign’s first-ever customer self-service mobile activation system, which won the 2003 SUPERQuest Award; and TAG at Microsoft, a production LLM orchestration platform that shipped nearly 100 Terraform articles by parsing XML templates, routing prompts to Azure OpenAI, and composing results with browser-driven Azure portal capture. Along the way: C++ MVP, nine technology books, 100+ articles for .NET Programming Tips & Techniques, and helping run CodeGuru, the largest Windows developer community of its era.

After 21 years at Microsoft, I’m now full-time on what’s next: completing my Master’s in Computer Science and publishing at Signal & Syntax, where I research and write about the mathematics and engineering underneath modern AI. I use mathematical models, experiments, and small implementations to explore everything from gradient descent and attention to inference-time behavior and training infrastructure; not just how to use these systems through an API, but how the machinery beneath that API actually works.

Connect with me on LinkedIn .