Probability & Linear Algebra for Programmers

Probability and Linear Algebra for Computing

Two branches of mathematics show up across modern software far more than calculus or proofs. Probability describes uncertainty: how often a hash collides, how many alerts are false alarms, how likely a system with three dependencies is to be up, how request arrivals cluster, and whether an A/B test result is real. Linear algebra describes collections of numbers and the operations on them: embeddings compared by cosine similarity, images and transformations in graphics, recommendation models, regression, PageRank, and the matrix multiplications at the heart of every neural network.

You do not need a mathematics degree to use either well. You need intuition for a dozen core ideas, the ability to do quick estimates, and awareness of the numerical traps that make correct formulas give wrong answers on a computer.

TL;DR

Conditional probability and Bayes' theorem explain why accurate detectors still produce mostly false alarms when events are rare.
Expectation and variance summarize random quantities; percentiles matter more than averages for latency.
Arrivals are often modeled as Poisson, gaps as exponential; latency and sizes often have long tails.
The birthday problem: collisions appear after about the square root of the space size — about 77,000 items for 32-bit IDs at 50%.
Vectors and dot products underlie similarity search; matrices represent transformations and datasets.
Prefer solving systems over computing inverses, and remember that floating-point arithmetic is approximate.

Quick Example

Three back-of-the-envelope calculations engineers make, in Python:

Each answer is surprising without the math and obvious with it.

Core Concepts

The probability ideas below come up constantly in engineering work; the linear algebra essentials follow in the next section.

Events and Conditional Probability

The probability of A given B, written P(A | B), is the probability of A when you know B happened: P(A and B) / P(B). Confusing P(A | B) with P(B | A) is one of the most common reasoning errors — "most attacks trigger an alert" is very different from "most alerts are attacks".

Bayes' theorem connects them: P(A | B) = P(B | A) × P(A) / P(B). The base rate P(A) dominates when events are rare, which is why the detector in the example produces 99 false alerts for each real one. See Security Detection Engineering.

Independence and Availability

Events are independent when one does not change the probability of the other. Independence makes availability math simple:

Real failures are often correlated — shared networks, deployments, regions — so redundancy delivers less than the independent calculation promises. See High Availability.

Random Variables, Expectation, and Variance

A random variable assigns numbers to outcomes. Its expected value is the long-run average; its variance (and standard deviation) measures spread. Linearity of expectation — the expected sum is the sum of expectations, even for dependent variables — makes many estimates easy, such as the expected number of collisions in a hash table.

Distributions Engineers Meet

For long-tailed quantities, averages are misleading: report percentiles such as p50, p95, and p99. See SLOs.

The Birthday Problem

With N possible values, collisions become likely after roughly √N random items — about 1.18 × √N for a 50% chance. That is why 32-bit random IDs are unsafe at modest scale, why hash tables need resizing, and why cryptographic hashes use 256 bits. See Distributed IDs.

Linear Algebra Essentials

Vectors, Dot Products, and Norms

A vector is an ordered list of numbers — a point, a direction, or a feature set. The dot product a · b sums element-wise products; dividing by the vectors' lengths (norms) gives cosine similarity, the standard measure for comparing text embeddings in semantic search. See Embeddings and Vector Databases.

Matrices as Data and Transformations

A matrix can hold a dataset (rows are examples, columns are features) or represent a linear transformation — rotation, scaling, projection — applied by matrix-vector multiplication. Composing transformations is matrix multiplication, which is why graphics pipelines and neural networks are built from it. Matrix multiplication is not commutative: AB usually differs from BA.

Solving Systems and Least Squares

Many problems reduce to solving Ax = b. When there are more equations than unknowns — fitting a line to noisy data — least squares finds the x minimizing the error, which is linear regression. Numerically, use solvers (np.linalg.solve, np.linalg.lstsq) rather than computing the inverse A⁻¹, which is slower and less accurate.

Eigenvectors and SVD

An eigenvector of A is a direction A only stretches. Eigenvectors explain PageRank (the dominant eigenvector of a link matrix), stability of iterative systems, and principal components. The singular value decomposition factors any matrix into rotations and scalings; it underlies PCA for dimensionality reduction, low-rank approximations for compression, and classic recommender systems. See Scikit-learn.

Numbers on Real Computers

Floating point represents most decimals approximately: 0.1 + 0.2 != 0.3. Compare with tolerances, and never store money in binary floats. See Binary Data and Character Encoding.
Cancellation: subtracting nearly equal numbers destroys precision; reformulate computations to avoid it.
Conditioning: some problems amplify input errors no matter the algorithm; check condition numbers for linear systems.
Logs for tiny probabilities: multiply many small probabilities by adding logarithms to avoid underflow.
Vectorize: libraries such as NumPy call optimized BLAS routines that are orders of magnitude faster than Python loops. See Python for Data Science.

Best Practices

Estimate Before You Build

Quick probability and capacity estimates — collision risk, false alarm rates, availability — catch design flaws earlier than any test. See Big O Notation for the complexity side of estimation.

Report Distributions, Not Just Means

Show percentiles and spread for latencies, costs, and experiment results. Means hide the tails users feel.

Question Independence

Before multiplying probabilities, ask what could make failures or events happen together.

Use Libraries for Numerical Work

Rely on well-tested numerical libraries for solving, decomposing, and sampling rather than implementing algorithms from textbooks.

Common Mistakes

Ignoring the Base Rate

Too-Small Random Identifiers

Inverting Matrices to Solve Equations

FAQ

Do software engineers need probability?

Yes, regularly: for estimating collisions and capacity, reasoning about failures and availability, interpreting monitoring and alerts, designing experiments, and understanding randomized algorithms and machine learning.

What linear algebra do programmers need?

Vectors, dot products, norms, matrix multiplication, solving linear systems, and an intuition for eigenvectors and SVD cover most needs in graphics, machine learning, search, and data analysis.

What is cosine similarity?

The dot product of two vectors divided by the product of their lengths. It measures how closely two vectors point in the same direction and is the standard way to compare embeddings.

What is the birthday problem in computing?

The observation that among N possible values, random collisions become likely after about the square root of N items. It sets minimum sizes for random identifiers and hashes.

Why does 0.1 + 0.2 not equal 0.3 in most languages?

Binary floating point cannot represent 0.1 or 0.2 exactly, so their sum is a nearby approximation that differs from the approximation of 0.3. Use tolerances for comparisons and decimal types for money.