Computer Architecture for Developers: CPUs & Caches

Q: How do I find out whether my program is memory-bound?

Run it under perf stat and look at instructions per cycle (IPC). An IPC well below 1 with high cache-miss counts suggests the core is waiting on memory. Profilers such as perf record, Intel VTune, or AMD uProf can attribute misses to specific lines. See Profiling & Diagnostics.

Computer Architecture

Computer architecture is the contract between software and the processor: which instructions exist, how they execute, and how the machine moves data between registers, caches, memory, and devices. Modern CPUs keep that contract simple on the surface — instructions appear to run one after another — while doing something far more elaborate underneath: executing many instructions at once, guessing which way branches go, and hiding memory latency behind several layers of cache.

For developers, the practical consequence is that instruction counts rarely explain performance; memory access patterns do. Two loops with identical big-O complexity can differ by an order of magnitude because one walks memory sequentially and the other jumps around. Understanding the machine turns these differences from mysteries into predictions.

TL;DR

A CPU core executes a pipeline of instructions out of order and in parallel, then commits results in program order so the illusion of sequential execution holds.
The memory hierarchy spans roughly 1 ns (L1 cache) to 100 ns (DRAM); a cache miss can cost as much as hundreds of simple instructions.
Caches move data in cache lines (typically 64 bytes), so sequential access is fast and scattered access is slow, regardless of how little data you actually use.
Branch prediction keeps pipelines full; unpredictable branches on hot paths cost 15–20 cycles each when the guess is wrong.
SIMD instructions process several values per instruction; compilers vectorize simple loops over contiguous arrays automatically.
Measure with hardware counters (perf stat) before optimizing — intuition about the machine is frequently wrong.

Quick Example

The same sum over the same matrix, in two traversal orders. Only the memory access pattern changes.

On a typical laptop the column-order loop is several times slower. Both loops perform exactly the same arithmetic; the difference is entirely cache and TLB behavior.

Core Concepts

The Instruction Set Architecture

The instruction set architecture (ISA) is the vocabulary a processor understands: registers, instructions, addressing modes, and the memory-ordering rules. The three ISAs most developers meet are:

Application code rarely depends on the ISA directly, but concurrent code does: a data race that never misbehaves on x86 can fail on ARM because the hardware reorders memory operations more freely. See Concurrency and Parallel Computing.

Pipelines and Out-of-Order Execution

A classic pipeline splits each instruction into stages — fetch, decode, execute, memory access, write-back — so several instructions are in flight at once. Modern cores go much further: they are superscalar (several instructions per cycle), out of order (independent instructions execute as soon as their inputs are ready), and use register renaming and a reorder buffer to commit results in program order.

The practical lesson: dependency chains are slow and independent work is fast. A loop where each iteration depends on the previous result (x = f(x)) cannot use the core's parallelism; a loop over independent elements can.

The Memory Hierarchy

The numbers vary by processor, but the ratios are stable: each level is roughly an order of magnitude slower than the one above. Performance work is mostly about keeping the working set in the fast levels.

Cache Lines, Locality, and Coherence

Caches transfer fixed-size cache lines — 64 bytes on x86-64 and most ARM cores, 128 bytes on some Apple designs. Reading one byte brings in its whole line, so:

Spatial locality — using neighbouring data — makes the rest of the line free.
Temporal locality — reusing data soon — keeps the line resident.
Hardware prefetchers detect sequential and strided patterns and fetch lines before you ask.

With several cores, each with private caches, a coherence protocol (MESI and its variants) keeps copies consistent. When one core writes a line, other cores' copies are invalidated. That is correct but expensive when cores repeatedly write different variables that happen to share a line — false sharing.

Branch Prediction and Speculation

To keep the pipeline full, the core predicts the outcome of every branch and executes speculatively down the predicted path. Correct guesses are nearly free; a misprediction flushes the pipeline and costs roughly 15–20 cycles. Predictors are excellent at regular patterns (loop exits, mostly-true checks) and poor at data-dependent random branches.

Speculation also has security implications: the Spectre family of attacks showed that speculatively executed instructions leave observable traces in caches, which is why browsers and kernels added mitigations that cost some performance.

Virtual Memory and the TLB

Programs use virtual addresses that the hardware translates to physical ones using page tables managed by the operating system. The translation lookaside buffer (TLB) caches recent translations. Scattered access across a large address space causes TLB misses on top of cache misses; huge pages (2 MB or 1 GB instead of 4 KB) reduce that pressure for large in-memory datasets.

I/O: Interrupts, DMA, and Buses

Devices such as NICs and SSDs connect over PCIe and move data with direct memory access (DMA) — writing straight into memory without the CPU copying each byte — then raise an interrupt when work completes. High-throughput systems reduce per-operation overhead by batching, polling instead of interrupting, and avoiding extra copies between kernel and user space.

What This Means for Your Code

Data Layout Beats Micro-Optimization

Choose layouts that match how hot loops read data. An array of structs keeps each record together; a struct of arrays keeps each field together, which is better when a loop touches only one or two fields.

Prefer Contiguous Collections

Arrays and vectors beat linked lists and trees for iteration, even when the theory says the costs are equal, because each pointer dereference can be a cache miss. Use contiguous structures by default and switch only when insertion patterns demand it.

Keep Hot Paths Predictable

Sorting data before a branchy loop, replacing branches with arithmetic, or partitioning input so similar items are processed together all help the branch predictor. Profile first: modern predictors handle most code well.

Let the Compiler Vectorize

SIMD units (AVX2 and AVX-512 on x86, NEON and SVE on ARM) process 4–16 values per instruction. Compilers auto-vectorize simple loops over contiguous arrays without aliasing ambiguity. Check with compiler reports such as -fopt-info-vec (GCC) or -Rpass=loop-vectorize (Clang) before reaching for intrinsics.

Respect NUMA on Large Servers

Multi-socket servers attach memory to each socket; accessing the other socket's memory is slower. For memory-heavy services, keep threads and their data on the same node (numactl, or let the runtime pin them) and measure remote-access ratios.

Best Practices

Measure With Hardware Counters

perf exposes the counters that explain performance: cycles, instructions per cycle, cache misses, and branch misses.

Low instructions-per-cycle with many cache misses means memory-bound code: fix layout and access patterns, not arithmetic.

Size Working Sets to the Cache

Process data in blocks that fit in L2 or L3 (tiling or blocking), especially in matrix, image, and join operations. Blocking a matrix multiply is the classic example of a 5–10x gain without changing the algorithm.

Separate Data Written by Different Threads

Give per-thread counters and queues their own cache lines, or aggregate locally and combine at the end.

Batch Work Across Boundaries

System calls, device I/O, and network round trips all have fixed per-call costs. Batching amortizes them, which is why buffered I/O and bulk APIs matter even on fast hardware.

Common Mistakes

False Sharing Between Threads

Chasing Pointers in Hot Loops

Optimizing Without Measuring

FAQ

Why is my code slower than its big-O complexity suggests?

Big-O counts operations and ignores their cost. On real hardware an operation that hits L1 cache is about a hundred times cheaper than one that waits for DRAM. Algorithms with good locality — scanning arrays, B-trees, blocked matrix code — often beat theoretically equivalent ones that chase pointers. See Big O Notation for the theory side.

What is a cache line and why should I care?

It is the unit the cache moves between memory levels, usually 64 bytes. Reading one field brings in its neighbors, so packing data you use together makes later reads free. It also explains false sharing: two threads writing different variables in the same line force the line to bounce between cores.

Does it matter whether I run on x86 or ARM?

For most application code, no — compilers and runtimes hide the ISA. It matters for concurrent code that relies on undefined memory-ordering behavior (ARM reorders more), for SIMD intrinsics, and for performance tuning, where cache sizes and core designs differ. Test on the architecture you deploy to.

How do I find out whether my program is memory-bound?

Run it under perf stat and look at instructions per cycle (IPC). An IPC well below 1 with high cache-miss counts suggests the core is waiting on memory. Profilers such as perf record, Intel VTune, or AMD uProf can attribute misses to specific lines. See Profiling & Diagnostics.

Are Spectre and Meltdown still relevant?

Yes, as background. Operating systems, browsers, and hypervisors ship mitigations, and new speculative-execution variants still appear. Application developers mainly need to keep platforms patched and avoid building their own sandboxes that run untrusted code in the same process.