Signal & Syntax

The Discrete Mathematics Hiding Inside LLMs

Tue, 31 Mar 2026 06:00:00 -0700

It's true what they say: discrete math is everywhere you look.

A recent LinkedIn post from Michael Palmer described how discrete mathematics is the foundation for how computers reason about problems. That thread got me thinking about just how many discrete math concepts show up inside systems that seem purely statistical. LLMs are often described in terms of neural networks, gradient descent, and probability distributions. If you’ve taken discrete mathematics and wondered what it has to do with modern AI, the answer is: more than you’d expect.

Underneath the calculus and linear algebra, the same structures you learn in a discrete math course keep appearing: sets, predicate logic, Boolean operations, modular arithmetic, formal proof patterns. This post traces those connections.

“Attention heads act like soft predicates over tokens. Masks are set operations. Chain-of-thought resembles proof structure.”

How Large Language Models (LLMs) Know Things They Were Never Taught

Mon, 09 Feb 2026 06:00:00 -0700

The model didn't learn this. It just read it.

When you ask an LLM without web search enabled a question like “What happened in the news this morning?”, the LLM will respond by telling you that it doesn’t have access to current events and suggest you check a more current news source such as Reuters or Google News.

Conversely, ask an LLM with web search enabled the same question, and you receive a detailed rundown of breaking stories, political controversies, and sports news from the past 24 hours.

Identical question. Same underlying technology. Completely different answers. The difference isn’t that one model is smarter or more current than the other. The difference is whether web search was triggered.

But why does that matter? Both models were trained months ago. Their internal knowledge stopped updating the moment training ended. So how does flipping a switch allow one model to suddenly “know” what happened this morning? The answer reveals a fundamental distinction most users never consider: the difference between what a model learned and what a model read.

LLMs don’t update their weights (the billions of numerical parameters that encode everything learned during training) when you chat with them. They don’t learn from your conversations. But they can access external information and reason over it within their context window. This isn’t learning; it’s reading. And understanding that difference changes how you think about what these systems can and cannot do.

“A model with web search doesn’t know more. It can see more. The knowledge lives in the retrieved text, not in the weights.”

Temperature and Top-P: The Creativity Knobs

Wed, 24 Dec 2025 12:00:00 -0800

Turn it up and things get weird.

Every API call to ChatGPT , Claude , or any other LLM includes two parameters most people either ignore or tweak randomly: temperature and top-p. The defaults work fine for casual use, so why bother understanding them? Because these two numbers fundamentally control how your model thinks.

The temperature value determines whether the model plays it safe or takes creative risks while the top-p value decides how many options the model even considers. Together, these values shape the personality of every response you receive.

I’ve watched developers cargo-cult settings from others without understanding what they do. “Set temperature to 0.7 for creative writing” becomes tribal knowledge, passed down without explanation. Let’s fix that by opening the hood and examining the mathematics that makes these knobs work.

“Temperature doesn’t make the model smarter or dumber. It changes how much the model trusts its own first instinct.”

The Wreck of the Edmund Fitzgerald: Modeling Decomposition in Extreme Environments

Sun, 30 Nov 2025 06:00:00 -0700

At 530 feet and 39°F, physics ensures Gordon Lightfoot's evocative lyric remains literally true.

Originally appearing on his 1976 album, Summertime Dream, “The Wreck of the Edmund Fitzgerald” is a powerful ballad written and performed by folk singer Gordon Lightfoot . In 1976, the song hit No. 1 in Canada on the RPM chart, and No. 2 in the United States on the Billboard Hot 100. The lyrics are a masterpiece, but there was one specific line that always stood out to me: “The lake, it is said, never gives up her dead.” Following the singer’s death in 2023, the song reconnected with older fans and reached new generations of listeners, making it to No. 15 on Billboard’s Hot Rock and Alternative category.

Listening to it again after all these years, I was inspired to research what that line meant and if there was any truth to it. What I discovered was very illuminating: Lake Superior really doesn’t give up her dead, and the science behind it is as haunting as the song itself.

It turns out that line is not poetic license. It’s physics.

When 29 souls went down with the Edmund Fitzgerald on November 10, 1975, they stayed down. Not because of some mystical property of the Great Lakes, but because of a perfect storm of temperature, pressure, and biology that we can model mathematically.

Note: This post contains scientific discussion of decomposition and forensic pathology in the context of maritime disasters.

“The lake, it is said, never gives up her dead / When the skies of November turn gloomy”

The Birthday Paradox in Production: When Random IDs Collide

Fri, 28 Nov 2025 06:00:00 -0700

With just 23 random selections from 365 options, you have a 50% chance of collision. Now imagine billions of UUIDs.

You generate a UUID. It’s 128 bits total, with 122 bits of randomness. That’s 340 undecillion possible values. Collision-proof, right? Your system generates a million IDs per second. Still safe? What about a billion?

As I like to say, common sense and intuition are the enemies of science. Common sense tells you that with 340,000,000,000,000,000,000,000,000,000,000,000,000 possible values, you’d need to generate at least trillions before worrying about duplicates. Maybe fill 1% of the space? 10%?

Math shows us the uncomfortable truth: You’ll hit a 50% collision probability after generating just $2.7 \times 10^{18}$ IDs. That’s 0.0000000000000000008% of your total space. At a billion IDs per second, you’ve got 86 years. Comfortable, but not infinite. Drop to 64-bit IDs? Now you’ve got 1.4 hours. Just enough time to duck out for long lunch and return to a disaster. And 32-bit? 77 microseconds. Faster than you can blink.

You might know that the birthday paradox proves that just 23 people have more than a 50% probability of sharing a birthday. What you may not know is that this isn’t just a party trick; it’s the same mathematics that determines when your “guaranteed unique” database IDs collide, why hash tables need careful sizing, and when your distributed system’s assumptions break.

“In a room of 23 people, there’s a greater than 50% chance two share a birthday. In your database, collisions arrive far sooner than intuition suggests.”

Hash Collisions: Why Your 'Unique' Fingerprints Aren't (And Why That's Usually OK)

Mon, 17 Nov 2025 06:00:00 -0700

In 2017, Google proved SHA-1 was broken by creating two different PDFs with identical hashes.

In 2017, Google researchers generated two different PDF files with identical SHA-1 hashes, finally proving what cryptographers had warned about for years: hash functions don’t create truly unique fingerprints ( Stevens et al., 2017 ). This “SHAttered” attack required 9 quintillion SHA-1 computations, which is the equivalent to 6,500 years of single-CPU computation. The attack cost approximately $45,000 in cloud computing resources, making it accessible to well-funded adversaries but not casual attackers.

Yet despite this proof, we still trust hash functions for everything from Git commits to blockchain transactions to password storage. The reason is simple: while collisions are mathematically inevitable, meaningful collisions remain virtually impossible. The full story of hash collisions is more nuanced than “unique” versus “not unique.”

“In cryptography, ‘secure’ has always meant ‘secure for now’.”

How Large Language Models (LLMs) Tokenize Text: Why Words Aren't What You Think

Tue, 11 Nov 2025 06:00:00 -0700

LLMs read tokens. Not words. A distinction with a technical—and potentially financial—difference.

When you type “I love programming” into ChatGPT, you might assume the model reads three words. It doesn’t. It reads somewhere between three and seven tokens, depending on how the text is split.

When you ask Claude to count the letters in the word “strawberry,” it often gets it wrong. The reason is simple. Claude never saw the word “strawberry” as a complete unit. It saw tokens like "str", "aw", "berry" and tried to reason about letters it couldn’t directly access.

And when early GPT-3 users discovered that typing “SolidGoldMagikarp” caused the model to behave erratically - generating nonsense, refusing requests, or producing bizarre outputs - the culprit wasn’t the model’s training. It was a glitch token: a tokenization artifact that never appeared in training data, leaving the model with no learned representation for how to handle it ( Rumbelow & Watkins, 2023 ).

“To a language model, text isn’t a stream of words. It’s a sequence of tokens. The way those tokens are created determines what the model can and cannot understand.”

How Large Language Models (LLMs) Handle Context Windows: The Memory That Isn't Memory

Mon, 10 Nov 2025 06:00:00 -0700

Context windows create the illusion of memory through mathematical attention, not storage.

When you have a long conversation with a large language model (LLM) such as ChatGPT or Claude , it feels like the model remembers everything you’ve discussed. It references earlier points, maintains consistent context, and seems to “know” what you talked about pages ago.

But here’s the uncomfortable truth: the model doesn’t remember anything. It’s not storing your conversation in memory the way a database would. Instead, it’s rereading the entire conversation from the beginning every single time you send a message.

“A context window isn’t memory. It’s a performance where the model rereads its lines before every response.”

Rethinking the Three-Second Traffic Rule: When Physics Says It’s Not Enough

Thu, 23 Oct 2025 09:00:00 -0700

Following distance isn't just space, it's time in motion.

While researching why car insurance rates are so extremely high in Las Vegas, I started thinking about the three-second rule and its validity. As I’ve always heard, the three-second rule refers to how far you should be behind a car in traffic. The idea is that you pick out a fixed roadside marker and you are supposed to pass that marker at least three seconds after the car in front of you. That rule is simple enough, yet deceptively deep once you unpack the physics.

“Three seconds is a rule of thumb. Physics reveals the truth.”

Modeling Heat Capacity and Evaporation with Python: Why Water Warms Slowly but Cools Fast

Sun, 12 Oct 2025 09:00:00 -0700

Water warms slowly but cools fast because high heat capacity resists change while evaporation accelerates night losses.

Every summer, it feels like a small miracle when the pool finally warms up enough to swim. In Nevada, where the air temperature can sit above 100°F (38°C) for weeks, you’d expect the water to keep pace. Yet, somehow, it takes forever to warm, and only a few cool nights can undo all that progress.

The same phenomenon shows up in a stick of butter. Butter melts quickly, while margarine stays stubbornly firm even under the same heat. That’s not coincidence; it’s thermodynamics.

The butter versus margarine comparison is a staple example in nutrition science. It shows how the proportions of fat, water, and solids affect how much energy it takes to change temperature. Butter, with more fat and less water, heats up and melts quickly. Margarine, full of water and unsaturated oils, absorbs more energy before softening because water’s specific heat is much higher.

“A pool in the desert and a stick of margarine in the kitchen both tell the same story: water resists change.”

How Large Language Models (LLMs) Learn: Calculus and the Search for Understanding

Wed, 08 Oct 2025 06:00:00 -0700

Meaning takes shape as the model learns to descend its own mathematical terrain.

When you interact with a large language model (LLM) such as ChatGPT or Claude , the model seems to respond instantly relative to the question’s degree of difficulty. What’s easy to forget is that every word it predicts comes from a long history of learning where billions of gradient steps have slowly sculpted its understanding of language.

Large language models don’t memorize text. They optimize it. Behind that optimization lies calculus. I’m not referring to the calculus you did with pencil and paper. I’m talking about a sprawling, automated version that computes millions of derivatives per second.

At its heart, every LLM is a feedback system. It starts with random guesses, measures how wrong it was, and then adjusts itself to be slightly less wrong. The word “slightly” in this context is the essence of calculus.

“Each gradient step represents a measurable reduction in error, guiding the model toward a more stable understanding of language.”

How Large Language Models (LLMs) Think: Turning Meaning into Math

Tue, 07 Oct 2025 06:00:00 -0700

Meaning takes shape in mathematics long before it reaches words.

When you enter a sentence into a Large Language Model (LLM) such as ChatGPT or Claude , the model does not process words as language. It represents them as numbers.

Each word, phrase, and code token becomes a vector — a list of real-valued coordinates within a high-dimensional space. Relationships between meanings are captured not by grammar or logic but by geometry. The closer two vectors lie, the more similar their semantic roles appear to the model.

This is the mathematical foundation of large language models: linear algebra. Matrix multiplication, vector projection, cosine similarity, and normalization define how the model navigates this vast space of meaning. What feels like understanding is actually the alignment of high-dimensional vectors governed by probability and geometry.

“Linear algebra and geometry do more than support AI; they create its language of meaning.”

How Large Language Models (LLMs) Read Code: Seeing Patterns Instead of Logic

Mon, 06 Oct 2025 09:00:00 -0700

AI reads code as patterns, not instructions.

Developers are accustomed to thinking about code in terms of syntax and semantics, the how and the why. Syntax defines what is legal; semantics defines what it means. A compiler enforces syntax with ruthless precision and interprets semantics through symbol tables and execution logic. But a Large Language Model (LLM), reads code the way a seasoned engineer reads poetry, recognizing rhythm, pattern, and context more than explicit rules.

“When an AI system ‘understands’ code, it is not executing logic; it is modeling probability.”

Numeric Parsing in Python with Integer Division and Modulus

Wed, 17 Sep 2025 13:00:00 -0700

Parse fixed-width numbers with // and % for speed and clarity.

When you need to parse a number, the first instinct is often to convert it to a string and slice it. That works well for data that comes from people — like phone numbers, credit cards, or postal codes — where formatting and leading zeros matter. But when you are working with raw numeric data that is guaranteed to be fixed-width and free of formatting, numeric parsing with integer division (//) and modulus (%) is the better option.

String parsing is flexible, but numeric parsing is faster and cleaner when the data is truly numeric.

Using SymPy in Python When NumPy Isn't Enough

Fri, 05 Sep 2025 11:45:00 -0700

SymPy provides symbolic computation and precision techniques for Python developers

Most of us reach for NumPy whenever math shows up in a project. But sometimes, you don’t want approximate answers, you want exact math. That’s when you pull SymPy out of your programmer’s toolkit and get to work.

It’s easy to think of SymPy only in academic terms, like running physics simulations where small rounding errors can snowball into nonsense, or checking algebraic identities where a value such as 0.0000001 should really be treated as exactly 0. Those are valid use cases, but they barely scratch the surface.

In real-world business applications, imprecision can be just as costly. Financial software is the most obvious example, where a few pennies lost to rounding errors can add up to millions at scale. Supply chain and logistics systems can also suffer when tolerances or unit conversions drift slightly off, leading to incorrect shipments or mismatched inventory. Even common scenarios such as pricing models or tax calculations can go sideways if the math behind them is not exact.

“Floats guess. SymPy knows.”

This is where SymPy shines. To see the difference between floating-point approximations (Python or NumPy) and symbolic precision (SymPy), let’s look at a simple but very real example from finance.

The Five-Second Rule Explored with Math & Python

Thu, 04 Sep 2025 06:00:00 -0700

Forget the five-second rule. Germs transfer instantly, and we can prove it with math.

You know the story: drop a cookie on the kitchen floor, swoop in before five seconds are up, and declare it safe. It is comforting. It is also wrong.

“Germs don’t wait five seconds. They start the party the instant your food hits the floor.”

The truth is much more interesting than the myth. Germs do transfer gradually, but they are especially fast at the beginning. That means if you want to know whether your floor-cookie is still edible, you need to think in curves, not in timers. And curves are something we can model.

The Meeting Diet: An Optimization Approach to Your Calendar

Thu, 28 Aug 2025 05:00:00 -0700

If you don't optimize your schedule, your schedule will optimize you. Badly

Every week your calendar fills with more meeting invites than you can reasonably handle. Which ones are worth the time and energy, and which should you politely decline? What if there was a way to quantify that choice?

“Your calendar is a knapsack. Every meeting takes space, but only some add enough value to justify carrying them.”

The good news: math can help. By modeling your schedule as a 0/1 knapsack problem with two constraints , you can treat meetings like items with value, time cost, and energy cost. Classic optimization techniques then help decide which meetings to attend. In this post, we’ll walk through framing the problem, prompting AI to scaffold the code, and running a simulation to visualize your optimal “meeting diet.”

Using Python Dispatch Tables for Cleaner Validation

Fri, 22 Aug 2025 10:38:20 -0700

Dispatch maps turn a mess of conditionals into one elegant mapping of rules and results.

Let’s be honest: argument validation code is rarely the proudest part of anyone’s repo.

Most of us start with the usual suspects:

❌ The dreaded inverted-V tower of if/else statements
❌ A graveyard of guard clauses scattered line after line

Using a dispatch table for validation rules means: one dictionary, one loop, infinite sanity.

Both work fine… until they don’t. Then you’re left maintaining a wall of conditionals that feels like it was designed by a committee of goblins.

There’s a better way: dispatch tables!

From Ice Shows to Algorithms: Cracking the Truck-Packing Problem

Wed, 20 Aug 2025 09:00:00 -0700

How do you fit it all in?

My first full-time programming job was for Holiday on Ice, an international ice show. While I focused mainly on back office systems such as accounting, itinerary, and box office reporting, I knew that one of the biggest technical challenges faced by the show’s crew was efficiently loading trucks for the next city.

“Given the dimensions of a truck and a list of containers (with their dimensions and weight), in what order, position, and orientation should you pack the truck?”

One day, the controller asked me if I could code a system that took, as input, the trucks’ 3D dimensions and the 3D dimensions (and weight) of every object to be packed. Back in the Turbo Pascal era, exploring 3D packing was painful. Today, with Python and AI-assisted scaffolding, it’s surprisingly approachable.

From Solow to ChatGPT: Why Total Factor Productivity Can't Keep Up With Generative AI

Tue, 19 Aug 2025 05:00:00 -0700

Our tools for measuring progress were built for factories, not for AI.

If ChatGPT can write code, summarize legal briefs, and help draft business strategies in seconds, why doesn’t that show up in our productivity statistics?

Economists have long relied on a metric called Total Factor Productivity (TFP) to measure technological progress. But in an era of free digital tools and generative AI, TFP looks more like a rearview mirror than a windshield. It tells us a lot about the past, but almost nothing about where the economy is headed.

You can see the ~~computer age~~ generative AI everywhere but in the productivity statistics.

(Adapted from Robert Solow, 1987)

Should You Walk or Run in the Rain? The Puzzle That Sparked a Passion

Mon, 18 Aug 2025 09:00:00 -0700

To walk or to run. That is the question. Early in my programming career, I came across a coding challenge that stuck with me for many years: “If it’s raining, will you stay drier by walking or running through it?” At the time, I didn’t have the skillset or tools to simulate the problem properly. It became one of the first exercises that nudged me toward a lifelong fascination with modeling the real world through code.

About Signal & Syntax

Mon, 01 Jan 0001 00:00:00 +0000

Most writing about LLMs sits at one of two altitudes: high-level explainers aimed at executives and curious newcomers, or dense research papers aimed at other researchers. There’s less writing in the middle, where working engineers actually operate. Most engineers need to understand enough of what’s happening under the hood to build and debug real systems, but not so much that they feel like they’re replicating a PhD program. Signal & Syntax serves that middle altitude.

Behind the Code: Tom's Story

Mon, 01 Jan 0001 00:00:00 +0000

For forty years I’ve built platforms, not applications. Every time I’ve been asked to ship a product, the product has turned out to be a system for generating products: a metaprogramming layer, a runtime, a configuration-driven engine that outlives the feature request it started from. At AT&T I architected Forms Express, a four-layer dynamic runtime that let non-programmers create schema-agnostic insurance applications. At Auto Parts International I built a state-machine platform that drove an entire distribution system off data-queue records.