Data Retention & Deletion: Policies, Cascades & Crypto-Shredding

Data Retention & Deletion

"Delete this user's data" is one of the most deceptively hard requests in software. The DELETE statement is trivial; enumerating the copies is not. A single user's data typically exists in the primary database, its read replicas, last night's backup and the twenty before it, a search index, an analytics warehouse, a CDC stream sitting in Kafka, three log aggregators, an error-tracking service, a support tool, a CRM, an email provider, and a product analytics vendor.

A team that never asked this question ships a product where the honest answer to a legally binding erasure request is "we don't know." That's a compliance failure and, more practically, an operational one: you also can't answer "how much data are we actually storing about people who churned three years ago?"

The other half of the topic is the one nobody asks for. Retention — deleting data on a schedule because you no longer need it — is a legal obligation under GDPR's storage limitation principle, not a storage optimization. Data you keep is data you must protect, must include in every access request, and will lose in a breach.

TL;DR

Retention is an obligation, not a preference. Data must not be kept longer than its purpose requires.
Set a retention period at collection time, per field, enforced by an automated job.
Enumerate every store — deletion is only as complete as your data inventory.
Soft delete is not deletion. It's a hidden flag; the data is still there and still in scope.
Anonymization often beats deletion for transactional records — keep the row, remove the person.
Backups: define a retention window so deleted data ages out, and re-apply erasures on restore.
Crypto-shredding — per-subject encryption keys, destroyed on erasure — makes backup copies unreadable.
Legal holds override everything. Check them before deleting anything.

Quick Example

A declarative retention policy driving an automated deletion job:

Three properties make this workable: the policy is declarative (auditors can read it, and it doubles as documentation), it runs in dry-run by default (deleting the wrong million rows is not recoverable), and every rule carries its basis — the answer to "why this period?" that auditors always ask.

Core Concepts

The many copies

The last category is the one nobody plans for and everyone has. A CSV a colleague exported for a board deck two years ago is a copy of personal data, in scope for erasure, and untracked. The mitigations are access control on exports, audit logging of them, and a policy that discourages ad-hoc extraction.

Soft vs. hard delete

Soft delete is a useful product feature — an undo window, a recycle bin — and it is not compliance deletion. A soft-deleted user's data is still stored, still in scope for a subject access request, and still lost in a breach.

The workable pattern is soft delete then hard delete: mark for deletion, keep for a defined grace period (30 days is common, and covers accidental deletions), then irreversibly remove. Communicate the window to users.

Cascading deletion

Database foreign keys handle the first column and nothing beyond it. Every system outside the primary database needs an explicit deletion path, and the ones that get missed are always the same: search indexes, caches, object storage, the warehouse, and third-party processors.

The anonymized_subject HMAC is worth noting: it preserves the ability to group a former customer's orders for financial reporting while removing the ability to identify them, provided the key is well protected.

Anonymization instead of deletion

For transactional data, deletion is usually wrong and anonymization is usually right:

Anonymized data falls outside GDPR entirely, so this satisfies the erasure request while preserving revenue reporting, cohort analysis, and financial audit. The bar is that re-identification must be genuinely impossible — see the quasi-identifier discussion in PII & Sensitive Data Handling, because a "region + date + amount" record can be surprisingly identifying at low volumes.

Backups

The recurring hard question, with a settled practical answer.

The deletion ledger must live outside the backed-up database, or a restore erases the record of what was supposed to be erased.

Crypto-shredding

The strongest technical answer to the backup problem:

Costs: key management at scale (one key per subject is a lot of keys), no ability to query or index encrypted fields, and a hard dependency on the KMS being available and its own backups not retaining destroyed keys. Its legal sufficiency isn't universally settled, though it's widely accepted as a strong measure.

Legal holds

A legal hold — litigation, a regulatory investigation, a law enforcement request — suspends deletion for the affected data, overriding both retention schedules and erasure requests. Deleting data under hold is spoliation of evidence and carries serious consequences.

Every automated deletion path must check holds first. Building the hold mechanism after the first hold arrives means someone manually disables the retention job, which then stays disabled.

Best Practices

Set retention at collection time

Every new field gets a period and a basis when it's introduced, recorded in the schema alongside its classification. "We'll decide retention later" means retaining forever, which is the default outcome everywhere it isn't decided.

Make the retention policy declarative and readable

A data structure or config file listing dataset, period, action, and basis. It doubles as auditor documentation, it's reviewable in a pull request, and it prevents retention logic scattering into a dozen ad-hoc scripts.

Dry-run by default, always

Deleting a million rows you didn't intend to is not recoverable. Every retention job reports what it would delete, requires an explicit flag to act, and alerts when the count deviates significantly from the previous run.

Anonymize transactional records rather than deleting them

Keep the order, remove the person. It satisfies erasure, preserves financial reporting and analytics, and avoids the cascading-delete problems that come with removing rows other systems reference.

Test deletion end to end, regularly

Create a test subject, exercise them across every system, request erasure, then verify nothing remains anywhere. Run it as an automated test. Deletion paths break silently when a new data store is added and nobody updates the erasure code.

Bound backup retention

An indefinite backup archive means deleted data persists indefinitely. A defined window (30–90 days) with documented aging-out is what makes the "backups will catch up" argument credible.

Check legal holds before any deletion

In the retention job, in the erasure handler, and in any manual deletion tool. A single unchecked path is the one that deletes evidence.

Log every deletion

What was deleted, from which systems, when, and under which rule. Proving you completed an erasure request is as important as completing it, and a deletion ledger outside the primary database survives a restore.

Common Mistakes

Treating soft delete as deletion

Deleting from the primary only

No legal hold check

Indefinite backup retention

No retention policy at all

Untested deletion paths

FAQ

Do we have to delete from backups?

Not immediately, under the generally accepted interpretation. What you must show is that deleted data isn't actively used, that it will age out on a defined schedule, and that a restore re-applies pending erasures. Document the backup retention window explicitly. Crypto-shredding is the strongest technical answer if you need one, at the cost of significant key-management complexity.

How long should we keep data?

As long as the purpose requires, and no longer. Practical anchors: session data 30 days, security logs 90 days, financial records 7 years (tax law), audit logs 1–7 years depending on the regime, and product analytics 13–25 months (which is what major analytics vendors default to). The important part is having a documented basis per dataset — auditors ask "why this period?" and "it seemed reasonable" is a weak answer.

Is anonymization enough for an erasure request?

Usually yes, if it's genuine — irreversible, and resistant to re-identification through quasi-identifiers or auxiliary data. Anonymized data falls outside GDPR entirely. The failure mode is calling pseudonymization anonymization: hashing an email or replacing a name with a stable opaque ID leaves the data personal and still in scope. See PII & Sensitive Data Handling.

What about data in third-party systems?

They're processors acting on your behalf, and they're in scope. Each needs a data processing agreement obliging them to delete on your instruction, and your erasure pipeline needs to call their deletion API. Maintain the list of processors — it's required for your RoPA anyway — and include every one in the automated flow.

How do we handle deletion in an event-sourced system?

This is a genuine architectural tension: event sourcing assumes an immutable log, and erasure requires removal. The standard answers are crypto-shredding (encrypt personal data in events with a per-subject key, destroy the key), keeping personal data out of events entirely and referencing a mutable store, or rewriting the log with a redaction event and rebuilding projections. Design for it up front — retrofitting erasure onto an append-only log is genuinely hard. See CQRS & Event Sourcing.

What's a reasonable grace period before hard deletion?

30 days is the common choice, and it balances two real risks: accidental deletion (people do change their minds, and support requests to restore accounts are common) against holding data longer than necessary. Communicate the window clearly, allow cancellation during it, and make sure the hard delete actually fires — a grace period that never expires is just soft delete with extra steps.