Operating System Concepts: Processes & Memory

Operating System Concepts

An operating system turns one machine into many: it gives every program the illusion of its own processor, its own memory, and its own files, while actually sharing a handful of cores and one pool of RAM among hundreds of processes. It also enforces isolation, so a crashing browser tab does not take the database down with it.

Developers meet the operating system constantly, usually indirectly — a service killed for using too much memory, a write that vanished after a power cut, a server that slows down under thousands of threads, or a container that behaves differently from a laptop. Each of these is an OS concept showing through the abstraction. Knowing the mechanisms makes the behavior predictable and the fixes obvious.

TL;DR

A process is an isolated address space plus resources; a thread is a schedulable flow of execution inside a process that shares its memory.
System calls are the doorway into the kernel; each costs from hundreds of nanoseconds to microseconds, so chatty I/O adds up.
Virtual memory gives each process private addresses backed lazily by physical pages, with copy-on-write, swapping, and an out-of-memory killer as the last resort.
The scheduler time-slices runnable threads across cores; context switches cost microseconds plus cold caches.
write() returns when data reaches the page cache, not the disk — durability needs fsync.
A container is an ordinary process isolated with namespaces and limited with cgroups, not a virtual machine.

Quick Example

Create a process, run another program in it, and wait for it — the fork/exec/wait pattern behind every shell and process supervisor.

Run it under strace -f -e trace=process ./spawn to watch the clone, execve, and wait4 system calls the kernel actually performs.

Core Concepts

Kernel Mode, User Mode, and System Calls

The CPU runs the kernel in a privileged mode and applications in an unprivileged mode. Applications cannot touch hardware, other processes' memory, or kernel data directly; they ask the kernel through system calls such as read, write, open, mmap, and clone. Libraries wrap these calls (fopen, os.open, fs.readFile), so you rarely see them — but strace (Linux) or dtruss (macOS) shows exactly what a program asks the kernel to do.

Processes and Threads

Isolation and parallelism pull in different directions: processes isolate failures but communicate through pipes, sockets, or shared memory; threads communicate cheaply but corrupt each other freely. See Concurrency and Parallel Computing.

Scheduling and Context Switches

The scheduler decides which runnable thread uses each core and for how long. Modern kernels are preemptive: a thread runs until it blocks, its time slice ends, or a higher-priority thread wakes. Linux has used the Completely Fair Scheduler for many years and switched to the EEVDF scheduler in kernel 6.6; both aim to share CPU time fairly while keeping interactive tasks responsive.

A context switch saves one thread's registers and loads another's. The direct cost is a few microseconds; the hidden cost is that the incoming thread finds caches and the TLB full of someone else's data. Thousands of threads ping-ponging on a few cores spend more time switching than working — the reason event loops and thread pools exist.

Load average counts threads that are running or waiting to run (and, on Linux, those in uninterruptible I/O wait). A load average persistently above the core count means work is queueing.

Virtual Memory

Every process sees a private address space divided into pages (usually 4 KiB). Page tables map virtual pages to physical frames, and the CPU's TLB caches those mappings. Key behaviors:

Demand paging — memory is allocated lazily; the first touch of a page triggers a minor page fault that maps it.
Copy-on-write — fork shares pages between parent and child until one writes, so forking a large process is cheap until it mutates memory.
Major page faults — reading a page back from disk (swap or a memory-mapped file) costs milliseconds.
Overcommit — Linux usually grants more virtual memory than physically exists, betting that not all of it is used. When the bet fails, the OOM killer terminates a process to recover memory.

Two numbers describe a process: VSZ (virtual size, mostly irrelevant) and RSS (resident set size, physical memory actually in use).

Files, Descriptors, and the Page Cache

A program opens files, sockets, and pipes as file descriptors — small integers indexing a per-process table. Reads and writes go through the kernel's page cache: reads are served from RAM when possible, and writes land in RAM first and reach the disk later through write-back.

That makes I/O fast and durability subtle. write succeeding means the kernel has the data, not the disk. fsync(fd) forces the file's data and metadata to stable storage; after creating or renaming a file, the containing directory must also be fsynced for the new name to survive a crash.

I/O Models

Blocking I/O — the thread sleeps until data is ready. Simple, but one connection per thread does not scale to tens of thousands of connections.
Readiness-based non-blocking I/O — epoll (Linux) and kqueue (BSD, macOS) tell one thread which of many sockets are ready. This is the engine under Node.js, nginx, and most async runtimes.
Completion-based I/O — io_uring (Linux) and IOCP (Windows) accept batches of operations and report when they finish, cutting system-call overhead for high-throughput storage and networking.

Signals

Signals are asynchronous notifications delivered to a process: SIGTERM asks it to stop, SIGKILL stops it unconditionally, SIGINT comes from Ctrl+C, and SIGCHLD reports a child's exit. Services should handle SIGTERM by finishing in-flight work before exiting — see Linux Processes & Signals and Graceful Shutdown.

Containers: Namespaces and Cgroups

A container is a normal process with two kernel features applied:

Namespaces give it a private view of process IDs, network interfaces, mounts, hostnames, and users.
Control groups (cgroups) cap and account its CPU, memory, and I/O.

There is no guest kernel, which is why containers start in milliseconds — and why a container that exceeds its memory limit is killed by the same OOM mechanism as any process. See Docker.

Reading What the OS Is Doing

See Linux Performance Troubleshooting for a structured workflow.

Best Practices

Match the Concurrency Model to the Work

Use an event loop or async runtime for many concurrent I/O-bound connections, a bounded thread pool for blocking work, and processes when you need fault isolation or must sidestep a global interpreter lock.

Make Durable Writes Explicit

When losing a write would corrupt state — configuration files, queues, ledgers — write to a temporary file, fsync it, rename it into place, and fsync the directory. Databases do this for you; hand-rolled storage usually does not.

Budget Memory, Not Just CPU

Set container memory limits with headroom above observed peak RSS, watch for memory growth over days, and alert on OOM kills. A process killed by the kernel leaves no stack trace in its own logs.

Handle SIGTERM

Orchestrators send SIGTERM, wait for a grace period, then send SIGKILL. Stop accepting new work, drain in-flight requests, and flush buffers within that window.

Close What You Open

File descriptors are finite (ulimit -n). Leaking sockets or files eventually fails with "too many open files" under load, long after the code that leaked them ran.

Common Mistakes

Treating write() as Durable

Reading "Free" Memory as Available Memory

One Thread per Connection

Forgetting to Reap Child Processes

FAQ

What is the difference between a process and a thread?

A process owns an isolated address space and resources such as open files; a thread is an execution path inside a process that shares that memory with its sibling threads. Processes isolate failures and memory corruption; threads share data cheaply but can corrupt each other. Most servers combine both: a few processes, each with a pool of threads or an event loop.

Why does Linux show almost no free memory?

Because unused RAM is wasted RAM: the kernel fills it with cached file data that it can drop instantly when programs need memory. The available column in free -h estimates how much memory programs can actually get. Low available memory and rising swap activity are the real warning signs.

What happens when a machine runs out of memory?

The kernel first reclaims page cache and, if configured, swaps out idle pages. If that is not enough, the out-of-memory killer picks a process — usually the largest — and terminates it with SIGKILL. In containers the same happens when a cgroup exceeds its memory limit, even if the host has free RAM.

How expensive is a context switch?

A few microseconds of direct cost, plus the indirect cost of cold caches for the thread that runs next. That is negligible at hundreds of switches per second and significant at hundreds of thousands. High context-switch rates (vmstat column cs) with modest useful work usually mean too many threads or lock contention.

Is a container a virtual machine?

No. A virtual machine runs its own kernel on virtualized hardware; a container is a process on the host kernel with isolated views (namespaces) and resource limits (cgroups). Containers are lighter and faster to start but share the host kernel, so a kernel vulnerability affects every container on the host.