Stateful Kubernetes Backup & Restore

Stateful Kubernetes Backup and Restore

Stateless services on Kubernetes are easy to recover: redeploy from Git and they are back. Stateful workloads — databases, message brokers, search indexes, file stores running on persistent volumes — are different. Their value is the data, and a backup is only useful if it captures that data consistently and can be restored into a working application within your recovery objectives. Many teams discover during an incident that their nightly volume snapshots contain a database caught mid-write, that restored pods cannot find their volumes, or that nobody knows how long a restore takes.

This page covers what to back up, the difference between crash-consistent and application-consistent backups, the main tools, and how to prove restores work.

TL;DR

Protect four things: cluster resources (ideally from Git), persistent volume data, secrets and keys, and, for self-managed clusters, etcd.
CSI volume snapshots are crash-consistent — like pulling the power; databases need application-consistent backups.
Prefer database-native backups (WAL archiving, logical dumps) through operators for databases; use volume snapshots for simpler stateful apps.
Velero backs up Kubernetes resources and volumes on schedules, with hooks to quiesce applications.
Copy backups to another region or account with immutability, so one compromise cannot delete everything.
Run restore drills into a separate namespace or cluster and measure RPO and RTO.

Quick Example

Nightly Velero backups of a namespace, including volume snapshots, kept for 30 days — and a restore drill into a different namespace:

Backup hooks make a simple file-based application consistent by freezing its filesystem during the snapshot:

Core Concepts

What to Protect

Consistency Levels

Multi-volume applications — for example a database with separate data and log volumes — need coordinated snapshots or native backups; independent snapshots may capture each volume at a different moment.

Tools

CSI VolumeSnapshots: the Kubernetes API for storage-level snapshots, implemented by CSI drivers.
Velero: backs up Kubernetes resources to object storage, and volumes via CSI snapshots or file-system backup (Kopia), with schedules, retention, hooks, and namespace mapping on restore.
Database operators: CloudNativePG, Percona, and similar operators provide native backups with point-in-time recovery to object storage.
Commercial platforms: Kasten K10, Portworx, and cloud backup services add policy management, application awareness, and reporting.

Restore Considerations

Order matters: CRDs before custom resources, namespaces and storage classes before PVCs, PVCs before pods. Tools such as Velero handle ordering for resources they back up.
Storage classes and zones: restored volumes must land in a zone where pods can run; cross-cluster restores need compatible storage classes.
Identity and secrets: restored workloads need credentials that still work — rotated secrets or external secret references may need updating.
External dependencies: DNS entries, load balancer addresses, and cloud IAM bindings often live outside the cluster.
Partial restores: practice restoring a single namespace, a single table, or a single volume, not only whole clusters.

See Database Restore Drills and Recovery Targets.

Best Practices

Use Native Backups for Databases

Physical backups with log archiving give point-in-time recovery and consistency guarantees that volume snapshots cannot. Use snapshots as an additional layer, not the only one.

Store Backups Off-Cluster and Immutable

Write backups to object storage in another account or region with object lock or equivalent immutability, so a cluster compromise or accidental deletion cannot remove them. See Immutable Backups and Restore Verification.

Drill Restores on a Schedule

Restore into an isolated namespace or cluster regularly, run application checks against the restored data, and record how long it took.

Monitor Backup Health

Alert on failed or missing backups, partially failed volume snapshots, and backups older than your recovery point objective.

Common Mistakes

Snapshots of a Busy Database

Backups in the Same Blast Radius

Never Testing a Restore

FAQ

How do I back up a Kubernetes cluster?

Keep workload definitions in Git, back up cluster resources and persistent volumes with a tool such as Velero, use native backups for databases, back up etcd for self-managed clusters, and store everything off-cluster with immutability.

Are volume snapshots enough to back up a database?

Usually not on their own. Snapshots are crash-consistent and may not capture multi-volume databases consistently. Native database backups with log archiving provide reliable, point-in-time recovery; snapshots are a useful additional layer.

What does Velero back up?

Kubernetes resources from the API server, stored in object storage, and persistent volume data through CSI snapshots or file-system backups. It supports schedules, retention, hooks, and restores into different namespaces or clusters.

How do I restore to a different namespace or cluster?

With Velero, use namespace mappings on restore, or install Velero in the target cluster pointing at the same backup storage. Make sure storage classes, zones, secrets, and external dependencies are compatible.

How often should I test restores?

Regularly — at least quarterly for critical systems, and after significant platform changes. Each drill should measure restore time and verify data with application-level checks.