Glossary

A

Ablation

An experimental technique that removes or disables part of a model to determine how that component contributes to its behavior.

Abstract syntax tree (AST)

A tree representation of the grammatical structure of source code, with nodes representing constructs such as expressions, statements, and declarations.

Adam

An optimization algorithm that adapts learning rates for individual parameters using estimates of the first and second moments of their gradients.

Adipocere

A wax-like substance, sometimes called grave wax, that can form from body fat during decomposition in moist, low-oxygen conditions.

ALiBi

Attention with Linear Biases, a positional method that adds distance-dependent biases directly to attention scores rather than adding positional embeddings to token representations.

Anaerobic

Occurring in the absence of oxygen.

Approximate nearest neighbor search

A search technique that efficiently finds vectors close to a query vector without exhaustively comparing the query against every stored vector.

Argmax

The input or index at which a function reaches its maximum value, such as the token index having the highest logit.

Argon2

A memory-hard password hashing algorithm designed to resist GPU and other highly parallel password-cracking attacks.

Arrhenius equation

An equation describing how the rate of a chemical reaction changes with temperature, with reaction rates generally increasing as temperature rises.

Asymptote

A line or value that a function approaches increasingly closely without necessarily reaching it.

Attention

A mechanism that allows a model to determine how strongly one token should relate to other tokens when constructing a representation.

Autoregressive

Describes a model that generates a sequence one element at a time, with each prediction conditioned on previously available elements.

Axis-aligned

Oriented so that an object's edges remain parallel to the coordinate axes rather than being placed at arbitrary angles.

B

Backpropagation

An algorithm that propagates error information backward through a neural network to compute gradients for its trainable parameters.

Batch gradient descent

Gradient descent in which each parameter update is calculated using the entire training dataset.

Benign overfitting

A phenomenon in which an overparameterized model fits its training data extremely closely yet still generalizes well to unseen data.

BERT

A transformer-based language model architecture designed to learn bidirectional contextual representations of text.

BigDecimal

A Java class for arbitrary-precision decimal arithmetic, commonly used when exact decimal calculations are required.

Binary decision variable

A decision variable restricted to the values 0 and 1, commonly used to represent choices such as select or reject.

Birthday paradox

The counterintuitive result that collisions become likely after selecting only about the square root of the number of possible values.

Boolean algebra

A system of logic based on true and false values and operations such as AND, OR, and NOT.

Boyle's Law

The principle that, at constant temperature, the volume of a gas varies inversely with its pressure.

Brute-force attack

An attack that systematically tries many possible inputs, such as passwords or cryptographic keys, until a desired result is found.

Buoyancy

The upward force exerted by a fluid on an immersed object, opposing the object's weight.

ByT5

A byte-level variant of T5 that processes raw UTF-8 bytes instead of using a conventional subword tokenizer.

Byte Pair Encoding (BPE)

A tokenization method that repeatedly merges frequently occurring adjacent units to construct a vocabulary of common subword patterns.

Byte-level BPE

A form of Byte Pair Encoding that begins with byte values rather than characters, allowing any byte-encoded text to be represented without an unknown token.

C

Causal mask

An attention mask that prevents a token from attending to future positions in a sequence.

Chain rule

A calculus rule for computing the derivative of a composition of functions by multiplying derivatives through the sequence of operations.

Chain-of-thought prompting

A prompting technique that elicits intermediate reasoning steps before a model produces its final answer.

Character-level model

A language model that processes individual characters or similarly fine-grained units rather than conventional subword tokens.

Character-level tokenization

A tokenization strategy in which individual characters are treated as tokens rather than words or subwords.

Checksum

A value calculated from data and used to detect errors or unintended changes in that data.

Coefficient of friction

A dimensionless value describing the resistance to sliding between two surfaces, such as a tire and a roadway.

Coefficient of performance (COP)

The ratio of useful heating or cooling delivered by a system to the energy required to operate it.

Collision probability

The probability that at least two independently selected values from a finite space are identical.

Collision resistance

A property of a cryptographic hash function that makes it computationally infeasible to find two different inputs that produce the same hash.

Conduction

The transfer of thermal energy through direct molecular interaction without bulk movement of the material.

Conservation of energy

The principle that energy cannot be created or destroyed, only transferred or transformed from one form to another.

Constraint

A condition or limit that a valid solution to an optimization problem must satisfy.

Constraint matrix

A matrix containing the coefficients that describe how decision variables participate in the constraints of an optimization model.

Context vector

A vector representation that summarizes information relevant to the current context for use in subsequent model computations.

Context window

The amount of information a language model can consider at one time during inference.

Control volume

A defined region of space used to analyze the flow and conservation of quantities such as mass and energy.

Convection

Heat transfer involving the motion of a fluid, such as air moving across the surface of water.

Coreference

The relationship between expressions in text that refer to the same entity, such as 'Ada' and 'she.'

Cosine annealing

A learning rate schedule that decreases the learning rate smoothly according to a cosine-shaped curve.

Cosine similarity

A measure of similarity between two vectors based on the cosine of the angle between them.

Crypto-agility

The ability of a system to replace or upgrade cryptographic algorithms without requiring a fundamental redesign.

Cumulative probability

The running total of probabilities as outcomes are accumulated in a specified order.

D

Dataclass

A Python class designed primarily to store data, with common methods such as initialization automatically generated from declared fields.

Deceleration

Acceleration that reduces an object's speed, such as the negative acceleration produced when a vehicle brakes.

Declarative programming

A programming style that describes what should be accomplished rather than explicitly specifying each control-flow step used to accomplish it.

Dense embedding

A vector representation in which information is distributed across many continuous numerical dimensions rather than represented primarily by zeros.

Derivative

A measure of how quickly a function changes with respect to one of its variables, such as how a quantity changes over time.

Deterministic

Describes a model or process whose inputs completely determine its outcome, without randomness.

Dictionary

A Python mapping type that stores values as key-value pairs and provides access to each value through its associated key.

Differential testing

A testing technique that compares the outputs of multiple implementations or executions to identify unexpected differences.

Dimensional analysis

A technique for checking equations and conversions by tracking physical units and verifying that they combine consistently.

Discrete mathematics

The branch of mathematics concerned with distinct, countable structures such as sets, logic, graphs, integers, and combinatorial objects.

Discrete model

A mathematical or computational model that represents a system using distinct values, states, events, or time steps rather than continuously varying quantities.

Dispatch table

A data structure that maps keys or conditions to functions, actions, or other behavior, allowing selection without a long series of conditional statements.

Distributed system

A system whose components run on multiple networked computers and coordinate to perform a common task.

Diurnal cycle

A pattern that repeats approximately every 24 hours, such as daily changes in temperature, sunlight, humidity, or wind.

Dot product

An operation that multiplies corresponding components of two vectors and sums the results, producing a scalar that reflects their alignment.

Double descent

A phenomenon in which test error first decreases, then increases near the interpolation threshold, and decreases again as model capacity grows further.

E

Early stopping

A regularization technique that stops training when performance on validation data stops improving.

Eigenvalue

The scalar factor describing how much an eigenvector is stretched, compressed, or reversed by a linear transformation.

Eigenvector

A nonzero vector whose direction remains unchanged when a particular linear transformation is applied to it.

Embedding

A numerical vector that represents a token or other object in a multidimensional space.

Embodied technology

Technological improvement incorporated into physical or capital goods, such as more capable machinery, computers, or infrastructure.

Energy balance

An accounting of energy entering, leaving, and accumulating within a system according to conservation of energy.

Entropy

A measure of uncertainty or dispersion in a probability distribution, with higher entropy indicating a more broadly distributed probability mass.

Epsilon comparison

A technique for treating two floating-point values as equal when the difference between them is smaller than a chosen tolerance.

Evaporation

The process by which molecules at the surface of a liquid escape into the gas phase.

Exploding gradient

A training problem in which gradients grow excessively large during backpropagation, causing unstable parameter updates.

Exploration-exploitation tradeoff

The balance between choosing familiar high-confidence options and exploring less certain alternatives that may produce better or more varied results.

Exponential function

A function in which a variable appears in the exponent, commonly used to model processes that grow or decay at rates proportional to their current state.

F

Feedforward network

A neural network component that applies learned transformations independently to each token representation after the attention operation.

Finite difference

A numerical technique that approximates derivatives using differences between function values evaluated at nearby points.

Fixed-width data

Data whose components occupy predetermined and consistent numbers of positions or digits.

Flash Attention

An optimized attention algorithm that reduces memory traffic by computing attention in blocks, enabling faster and more memory-efficient exact attention.

Floating-point

A numerical representation that approximates real numbers using a fixed number of significant digits and an exponent.

Forward Euler method

A numerical integration method that estimates the next state of a system using its current value and current rate of change.

G

Generative AI

Artificial intelligence systems designed to generate new content, such as text, images, audio, video, or code, in response to input.

Genetic algorithm

An optimization technique inspired by biological evolution that evolves candidate solutions using operations such as selection, crossover, and mutation.

Glitch token

A token that produces unusual or unreliable model behavior because its learned representation is missing, weak, or otherwise anomalous.

Gradient

A vector of partial derivatives describing how a function changes with respect to its inputs or parameters.

Gradient descent

An optimization method that iteratively adjusts model parameters in the direction that reduces a loss function.

Greedy algorithm

An algorithm that repeatedly makes the locally preferred choice at each step without reconsidering earlier decisions.

Greedy decoding

A decoding strategy that selects the highest-probability token at each generation step.

Gross Domestic Product (GDP)

The monetary value of final goods and services produced within a country during a specified period.

Grouped-query attention

An attention variant in which groups of query heads share key and value heads, providing a compromise between multi-head and multi-query attention.

Guard clause

A conditional check placed near the beginning of a function or block that exits early when a required condition is not satisfied.

Guillotine split

A space-partitioning technique that divides a rectangular region completely along one dimension, producing smaller rectangular regions for subsequent placement.

H

Hallucination

Model-generated information that is unsupported by the available evidence or presented as factual despite being incorrect or fabricated.

Hash collision

A situation in which two different inputs produce the same hash value.

Hash function

A function that transforms input data of arbitrary size into a fixed-size value called a hash or digest.

Hash table

A data structure that uses a hash function to map keys to locations where their associated values can be stored and retrieved.

Heat capacity

The amount of energy required to raise the temperature of an entire object or system by one degree.

Heat sink

A material, object, or system that absorbs and stores or dissipates thermal energy.

Heuristic

A problem-solving strategy designed to find a useful solution quickly without guaranteeing that the solution is optimal.

High-dimensional space

A mathematical space containing many independent dimensions, allowing complex objects and relationships to be represented as vectors.

HMAC

A keyed message authentication mechanism that combines a cryptographic hash function with a secret key to verify message integrity and authenticity.

Human-AI capital

A proposed way of viewing productive capability as the combined skills of people and the AI systems that augment their work.

Human-machine collaboration

A way of working in which people and computational systems contribute complementary capabilities toward completing a task or achieving an objective.

I

Ideal gas law

An equation relating the pressure, volume, temperature, and amount of an idealized gas, commonly written as PV = nRT.

IEEE 754

The widely used technical standard that defines formats and behavior for floating-point arithmetic in computer systems.

In-context learning

The ability of a language model to adapt its behavior based on examples or patterns provided in its current context without updating its model parameters.

Induction head

An attention head involved in a circuit that recognizes repeated token patterns and uses an earlier occurrence to predict what is likely to follow the current occurrence.

Inference

The process of using a trained model to generate predictions or outputs from new input.

Integer division

Division that produces an integer quotient by discarding the fractional or remainder portion of the result.

Integer linear programming (ILP)

An optimization method in which the objective and constraints are linear and some or all decision variables are restricted to integer values.

Integral

A mathematical operation representing accumulation, such as the area under a curve, and serving as an inverse operation to differentiation.

Irradiance

The radiant power received by a surface per unit area, commonly measured in watts per square meter.

J

Jacobian

A matrix containing the partial derivatives of a vector-valued function with respect to its inputs.

K

Key

In attention, a learned vector representing information a token exposes for comparison with queries.

Kinematics

The branch of mechanics that describes motion using quantities such as position, velocity, acceleration, and time without considering the forces causing the motion.

Kinetic energy

The energy an object possesses because of its motion, proportional to its mass and the square of its velocity.

Knowledge graph

A structured representation of entities and the relationships among them, typically modeled as nodes connected by labeled edges.

KV cache

A cache of previously computed key and value vectors that lets a transformer reuse them while generating subsequent tokens.

L

Lambda function

A small anonymous function in Python defined with the lambda keyword, typically used for simple expressions that do not require a named function.

Large language model (LLM)

A neural network trained on large amounts of text to model language and generate or interpret sequences of tokens.

Latent heat of vaporization

The energy required to convert a unit mass of liquid into vapor without changing its temperature.

Layer normalization

A normalization technique that stabilizes neural network training by rescaling activations within a layer.

Learned tokenization

An approach in which useful text segmentation is learned as part of model training rather than being completely fixed beforehand.

Learning rate

A training hyperparameter that controls the size of each parameter update during optimization.

Learning rate schedule

A strategy for changing the learning rate over the course of training.

Linear algebra

The branch of mathematics concerned with vectors, matrices, linear transformations, and the relationships among them.

Linear transformation

A transformation of vectors that preserves vector addition and scalar multiplication and can be represented by matrix multiplication.

Local minimum

A point where a function has a lower value than nearby points but may not have the lowest value across the entire function.

Log probability

The logarithm of a probability, commonly used in machine learning because it makes very small probabilities easier to compute with and combine.

Log-normal distribution

A probability distribution for a positive-valued variable whose logarithm follows a normal distribution.

Logit

A raw numerical score produced by a model before it is converted into a probability by a function such as softmax.

Longwave radiation

Thermal electromagnetic radiation emitted by objects because of their temperature, primarily at infrared wavelengths.

Loss function

A function that measures how far a model's predictions are from the desired outputs, producing a value that training attempts to minimize.

Luhn algorithm

A checksum algorithm used to detect common errors in identification numbers such as credit card numbers.

M

Mamba

A sequence-modeling architecture based on selective state space models that can process sequences with linear rather than quadratic scaling.

Masking

A technique that prevents an attention mechanism from considering selected token positions, typically by modifying their attention scores so they receive effectively zero attention weight.

Matrix multiplication

An operation that combines matrices by computing dot products between rows and columns, forming the basis of many neural network computations.

Maximization

An optimization objective that seeks the feasible solution producing the largest possible value of an objective function.

Mechanistic interpretability

The study of how neural networks implement computations internally by identifying and explaining components, circuits, and algorithms within trained models.

Memory-hard function

A function deliberately designed to require substantial memory as well as computation, making large-scale parallel attacks more expensive.

Mini-batch gradient descent

A form of gradient descent that computes each parameter update using a small subset of the training data.

Model parameter

A numerical value learned during training that influences how a model transforms inputs into outputs.

Model weights

The numerical parameters learned during training that determine how a neural network transforms inputs and produces outputs.

Modular arithmetic

A system of arithmetic in which numbers wrap around after reaching a specified modulus, producing cyclic patterns.

Modulus

An arithmetic operation that returns the remainder after one integer is divided by another.

Momentum

An optimization technique that incorporates previous gradients into current parameter updates, helping smooth oscillations and maintain progress.

Monte Carlo simulation

A computational technique that repeatedly samples random inputs to estimate the range, likelihood, or statistical properties of possible outcomes.

Monte Carlo simulation

A computational technique that uses repeated random sampling to estimate probabilities, outcomes, or other properties of a system.

Multi-head attention

An attention architecture that runs multiple independently learned attention heads in parallel and combines their outputs, allowing different relationships to be modeled simultaneously.

Multi-objective optimization

Optimization involving two or more objectives that may compete with one another, requiring trade-offs among possible solutions.

Multi-query attention

An attention variant in which multiple query heads share a single set of key and value heads, reducing memory use and inference cost.

Multidimensional knapsack problem

A generalization of the knapsack problem in which each item consumes multiple limited resources and the goal is to maximize total value without exceeding any resource constraint.

Multimodal tokenization

The conversion of information from multiple modalities, such as text, images, or audio, into discrete or structured units that a model can process.

N

Normalization

The process of rescaling numerical values or vectors to a standardized range or magnitude.

NP-hard

A class of computational problems at least as difficult as the hardest problems in NP, for which no efficient algorithm is known that always finds an optimal solution.

Nucleus sampling

Another name for top-p sampling, in which token selection is restricted to the smallest set of likely tokens whose cumulative probability reaches a chosen threshold.

Numerical stability

The ability of a numerical computation to avoid excessive errors caused by finite-precision arithmetic, overflow, underflow, or rounding.

O

One-hot

A representation or distribution in which one position has value 1 and all other positions have value 0.

Optimization

The process of finding the best solution to a problem according to a defined objective while satisfying specified constraints.

Optimizer

An algorithm that updates model parameters during training using gradients or related information to reduce a loss function.

Orchestration layer

The software layer surrounding a model that coordinates tasks such as retrieval, tool calls, context construction, and the flow of information between components.

Out-of-vocabulary (OOV)

Describes an input unit that is not present in a model's predefined vocabulary and therefore cannot be represented directly.

Overfitting

A condition in which a model fits its training data very well but performs poorly on new, unseen data.

P

Padding mask

An attention mask that prevents padding tokens added for sequence alignment from influencing model computations.

Password salt

Random data combined with a password before hashing so that identical passwords do not produce identical stored hashes.

Pigeonhole Principle

A mathematical principle stating that if more objects are placed into fewer containers, at least one container must contain more than one object.

Poisson process

A probabilistic model for events that occur independently and randomly over time or space at a specified average rate.

Positional encoding

Information added to token representations so a transformer can distinguish token positions and reason about sequence order.

Predicate logic

A formal system of logic that uses predicates, variables, and quantifiers to express statements about objects and their relationships.

Principal component analysis (PCA)

A dimensionality-reduction technique that identifies directions in data that account for the greatest variance.

Probability distribution

A mathematical description of the possible outcomes of a random process and the probabilities associated with those outcomes.

Probability mass

The amount of probability assigned to an outcome or collection of discrete outcomes.

Productivity paradox

The apparent disconnect between widespread adoption of new technologies and the absence or delay of corresponding gains in measured productivity.

Prompt engineering

The practice of designing and structuring model inputs to improve the relevance, reliability, or format of generated outputs.

PuLP

An open-source Python library for defining and solving linear and integer optimization problems.

Q

Q10 temperature coefficient

A measure of how much the rate of a biological or chemical process changes for each 10°C change in temperature.

Quadratic attention cost

The property of standard self-attention in which computation and attention-score storage grow approximately with the square of sequence length.

Query

In attention, a learned vector representing what a token is looking for when determining which other tokens are relevant.

R

Rainbow table

A precomputed data structure used to recover passwords by efficiently matching stored password hashes against previously calculated values.

Random seed

A value used to initialize a pseudorandom number generator so that the same sequence of random choices can potentially be reproduced.

Rate constant

A parameter that determines how quickly a process changes over time in a mathematical rate model.

Rational number

A number that can be expressed exactly as the ratio of two integers, such as 5/100.

Reaction time

The elapsed time between perceiving an event and beginning a physical response to it.

Regularization

Techniques that discourage a model from fitting training data too narrowly, improving its ability to generalize to unseen data.

Residual connection

A shortcut connection that adds a layer's input to its output, helping information and gradients flow through deep neural networks.

Residual stream

The evolving vector representation passed through transformer layers and updated by components such as attention and feedforward networks.

Retrieval-Augmented Generation (RAG)

A technique that retrieves relevant information from an external source and supplies it as context to a generative model.

RMSProp

An optimization algorithm that adjusts parameter learning rates using a moving average of squared gradients.

Root

A value of a variable that makes a mathematical function or equation equal to zero.

RWKV

A neural network architecture that combines characteristics of recurrent neural networks and transformers while supporting linear-time sequence processing.

S

Saddle point

A point where the gradient is zero but which is not a local minimum or maximum because the function rises in some directions and falls in others.

Sampling

The process of selecting an outcome randomly according to a probability distribution, such as choosing the next token during language generation.

Scaled dot-product attention

An attention mechanism that compares queries with keys using dot products, scales the resulting scores by the square root of the key dimension, applies softmax, and uses the resulting weights to combine values.

Self-attention

An attention mechanism in which tokens in a sequence determine the relevance of other tokens in that same sequence.

Semantic retrieval

Retrieval based on similarity of meaning rather than exact matching of words or phrases.

SentencePiece

A language-independent tokenization system that learns subword units directly from raw text without requiring pre-tokenization into words.

Sequence length

The number of tokens or other units in an input or output sequence.

Session token

A value used to identify and associate a client with an authenticated or otherwise established application session.

Set theory

The branch of mathematics that studies collections of objects and operations such as union, intersection, complement, and membership.

SHA-1

A cryptographic hash function that produces a 160-bit hash and is now considered insecure because practical collision attacks have been demonstrated.

Sharding

A database partitioning technique that distributes portions of a dataset across multiple independent servers or database instances.

Shortwave radiation

Electromagnetic radiation at relatively short wavelengths, including most of the solar energy reaching Earth's surface.

Simulated annealing

A probabilistic optimization algorithm that sometimes accepts worse intermediate solutions to escape local optima while gradually reducing that behavior during the search.

Simulation

The use of a computational model to imitate the behavior of a system so that its outcomes can be explored or analyzed.

Singular value decomposition (SVD)

A matrix factorization technique that decomposes a matrix into components representing important directions and their relative strengths.

Sliding window attention

An attention pattern that restricts each token to attending to a limited neighborhood of nearby tokens rather than the entire sequence.

Softmax

A function that converts a set of numerical scores into nonnegative values that sum to one, allowing them to be interpreted as a probability distribution.

Solow residual

The portion of economic output growth not explained by measured growth in labor and capital, commonly associated with changes in productivity or technology.

Specific heat

The amount of energy required to raise the temperature of a unit mass of a substance by one degree.

Standard deviation

A measure of spread equal to the square root of variance and expressed in the same units as the measured values.

Stateless

Describes a system in which each request is handled independently without retaining information from previous requests.

Static analysis

The examination of source code for properties, errors, or vulnerabilities without executing the program.

Stefan-Boltzmann law

A physical law stating that the thermal radiation emitted by an idealized surface is proportional to the fourth power of its absolute temperature.

Stochastic

Involving randomness or probability rather than producing a completely deterministic outcome.

Stochastic gradient descent (SGD)

A gradient-based optimization method that estimates gradients from randomly selected training examples rather than the entire dataset.

Stochasticity

The presence of randomness or probabilistic variation in a process, model, or system, causing the same conditions to potentially produce different outcomes.

Stopping distance

The total distance traveled from recognizing the need to stop until the object comes to rest, including both reaction and braking distance.

Subword tokenization

A tokenization strategy that represents text using units larger than individual characters but often smaller than complete words.

Surface area

The total area of the exposed surfaces of a three-dimensional object.

Symbolic algebra

The manipulation of mathematical expressions containing symbols, including operations such as expansion, factoring, simplification, and equation solving.

Symbolic computation

Mathematical computation that manipulates expressions and exact values symbolically rather than approximating them numerically.

Syntax and semantics

Syntax defines the valid structure of expressions in a language, while semantics defines what those expressions mean.

T

Temperature

A sampling parameter that rescales a model's logits before softmax, controlling how concentrated or spread out the resulting token probabilities are.

Thermal stratification

The formation of layers within a fluid at different temperatures, often with warmer, less dense fluid above cooler, denser fluid.

Thermocline

A layer in a body of water where temperature changes rapidly with depth, separating warmer surface water from colder deep water.

Thermodynamics

The branch of physics concerned with energy, heat, work, temperature, and the ways energy is transferred and transformed.

Thought experiment

A hypothetical scenario used to explore the consequences of assumptions or ideas through reasoning rather than direct experimentation.

Three-second rule

A driving guideline that recommends maintaining at least three seconds of travel time between your vehicle and the vehicle ahead.

Time headway

The time interval between two moving vehicles passing the same point, commonly used to express following distance independently of speed.

Token

A unit of text processed by a language model, such as a word, part of a word, punctuation mark, or other character sequence.

Token ID

The numerical identifier assigned to a token in a tokenizer's vocabulary.

Tokenization

The process of dividing input into tokens that a model can represent and process numerically.

Tool use

The ability of a model to request actions from external systems, such as web search, code execution, calculators, databases, or APIs, and then reason over the returned results.

Top-k

The k highest-ranked results selected from a larger set, such as the most relevant document chunks returned by a retrieval system.

Top-k sampling

A token-sampling strategy that restricts the model's next-token choices to the k tokens with the highest probabilities.

Top-p

A sampling parameter that restricts token selection to the smallest set of highest-probability tokens whose cumulative probability reaches a specified threshold.

Top-p (nucleus) sampling

A token-sampling strategy that selects from the smallest set of highest-probability tokens whose cumulative probability reaches a specified threshold p.

Total Factor Productivity (TFP)

An economic measure of the portion of output growth that cannot be explained by increases in measured inputs such as labor and capital.

Transformer

A neural network architecture built around attention mechanisms for processing relationships among elements of a sequence.

Trigonometry

The branch of mathematics concerned with relationships among angles and lengths, commonly expressed using functions such as sine, cosine, and tangent.

U

UTF-8

A variable-length Unicode character encoding that represents text using sequences of one to four bytes.

UUID version 1

A UUID format generated primarily from a timestamp and node identifier rather than random bits.

UUID version 4

A 128-bit UUID format in which 122 bits are randomly generated, providing an extremely large space of possible identifiers.

V

Value

In attention, a learned vector containing the information contributed by a token after its relevance has been determined.

Vanishing gradient

A training problem in which gradients become extremely small as they propagate backward through a deep neural network, slowing or preventing learning in earlier layers.

Variance

A measure of how widely numerical values are spread around their mean, equal to the average squared deviation from that mean.

Vector

An ordered list of numerical values that can represent an object, direction, or position in a multidimensional space.

Vector database

A database designed to store and search high-dimensional vectors, commonly used to retrieve items with embeddings similar to a query.

Vector norm

A numerical measure of a vector's magnitude or length.

Vector projection

The component of one vector that lies in the direction of another vector.

Vectorization

The technique of applying an operation to an entire array of values at once rather than processing individual elements with explicit loops.

Velocity

A vector quantity describing both the speed and direction of an object's motion.

Vocabulary

The complete set of tokens that a tokenizer can represent directly, with each token assigned a numerical identifier.

Volume utilization

The proportion of a container's available three-dimensional volume occupied by packed items.

W

Warm restarts

An optimization strategy that periodically raises the learning rate after it has decreased, allowing training to explore new regions of parameter space.

WordPiece

A subword tokenization algorithm that builds a vocabulary of reusable word pieces and is used by models such as BERT.

Working memory

Short-term information kept immediately available for ongoing reasoning or processing rather than stored as durable long-term memory.