PII & Sensitive Data Handling

Every privacy obligation depends on one prerequisite: knowing what personal data you hold and where it lives. You cannot encrypt, restrict, retain, delete, or report on data you haven't inventoried — and most organizations, when they finally do the inventory, discover personal data in places nobody expected. Application logs. A CSV in a shared drive from a 2023 migration. A notes free-text field that customer support has been pasting phone numbers into for two years. The analytics warehouse, where the "anonymized" user ID is joinable back to an email.

The second thing worth internalizing early is that "PII" is a fuzzier category than it sounds. GDPR's "personal data" covers anything relating to an identifiable person — which includes IP addresses, device identifiers, cookie IDs, and location traces. Data that identifies nobody in isolation frequently identifies someone in combination: a famous result showed that ZIP code, date of birth, and sex uniquely identify a large majority of the US population.

The engineering response is a small set of durable practices: classify at the schema level, encrypt by default, tokenize the highest-risk fields, mask everywhere data leaves the primary store, and minimize relentlessly.

TL;DR

Quick Example

Classification driving protection automatically, rather than by convention:

The value is that classification is declared once, in the schema, and every consumer — the logger, the retention job, the data inventory report, the test-data generator — derives its behavior from it. Redaction that depends on developers remembering at each call site fails within a month.

Core Concepts

What counts as personal data

The quasi-identifier column is where teams underestimate risk. Individually harmless, combined they identify. Latanya Sweeney's well-known result found that ZIP code, date of birth, and sex uniquely identify roughly 87% of the US population — which means a "de-identified" dataset with those three columns is not de-identified.

Anonymization vs. pseudonymization

This distinction is regularly misunderstood, usually in the direction of claiming anonymization for something that's pseudonymization. Hashing an email is pseudonymization: the space of email addresses is small enough to brute-force, so sha256(email) is reversible in practice. Even a random opaque ID is pseudonymous while the mapping exists anywhere.

Genuine anonymization requires removing or generalizing enough that re-identification is impossible even with auxiliary data — k-anonymity, l-diversity, or differential privacy. It's a research-grade problem, and aggregate statistics with suppressed small cells is the practical form most organizations achieve.

Encryption layers

The most common gap is treating at-rest encryption as sufficient. It protects against a lost disk and does nothing against the far more likely scenario: an application vulnerability, a leaked credential, or an engineer running an unrestricted query. Field-level encryption for the highest-sensitivity columns is what addresses that.

Tokenization

Tokenization is the strongest scope reduction tool available. A stolen database of tokens is worthless; the tokens have no mathematical relationship to the underlying values and cannot be reversed without the vault. For card data specifically it's the difference between your whole application being in PCI DSS scope and only a small isolated component being in scope.

Format-preserving tokens (same length and character class as the original) let you tokenize without schema changes, which makes retrofitting feasible.

Access control

Access to personal data should be least privilege, time-bound, and logged. Standing production database access for every engineer is the pattern most audits flag and most breaches exploit — replace it with a just-in-time access request that grants a short-lived role and writes an audit record. See Row-Level Security.

Logs and lower environments

Redaction must live in the logging layer, applied by default, with an explicit opt-out. Relying on developers to remember at each call site guarantees leakage — and log data is replicated to aggregators, retained for months, and accessible to a much wider group than the database.

For lower environments, the only reliable answer is never copy production personal data. Generate synthetic data, or apply irreversible masking during export. "We'll be careful with the staging database" is not a control.

Best Practices

Classify in the schema, not in a spreadsheet

A column comment, a dataclass annotation, or a dbt meta field. Classification that lives beside the data stays current and can drive tooling; classification in a document is stale within a quarter and drives nothing.

Run automated discovery periodically

Scan databases, object storage, and logs for patterns matching emails, phone numbers, card numbers, and national IDs. Tools like AWS Macie, Google DLP, or open-source scanners find the CSV in a bucket and the free-text field nobody classified. Run it on a schedule — new leaks appear continuously.

Redact at the logging layer

A logging filter or serializer that masks classified fields by default, with an explicit opt-out for the rare case. This is the single highest-value control against the most common leak path.

Never use production personal data in lower environments

Synthetic generation (Faker, Synth, or a domain-specific generator) or irreversible masking on export. A staging environment with real customer data has production risk and pre-production controls, which is the worst combination available.

Tokenize the highest-risk fields

Card data, national IDs, and bank details. Holding a token instead of the value removes an entire class of risk and dramatically reduces compliance scope. For cards specifically, use the payment provider's vault and never touch the raw number.

Encrypt fields the application doesn't need to query

Field-level encryption for health notes, government IDs, and free-text that may contain anything. The tradeoff is losing indexing and search on that column, so apply it where you genuinely only read the field by primary key.

Make production access just-in-time and logged

No standing access. A request, an approval, a short-lived credential, and an audit record of the query. This is both a strong control and the evidence an auditor will ask for.

Keep personal data out of URLs

Query strings appear in server access logs, browser history, Referer headers sent to third parties, and analytics. Use POST bodies or opaque path identifiers, and never put an email, a token, or a session identifier in a URL.

Minimize relentlessly

The cheapest control is not collecting the field. Review data collection periodically and ask what each field is actually used for — most systems accumulate fields nobody reads. Deleting a column removes every obligation attached to it at once.

Common Mistakes

Logging entire objects

Calling hashed data anonymous

Copying production data to staging

Personal data in URLs

Relying only on at-rest encryption

Standing production access for everyone

FAQ

Is an IP address personal data?

Under GDPR, yes — the CJEU held that a dynamic IP address is personal data for a party that has the legal means to identify the subscriber. In practice, treat IP addresses as personal data: include them in your inventory, set a retention period (30–90 days is common for security logs), and don't retain them indefinitely in analytics. US state laws vary, and the safe default is to treat them as personal.

Can we use production data for testing if we mask it?

Only if the masking is irreversible and applied during export, never afterward in the target environment. The risks are that masking misses a free-text field containing personal data, that quasi-identifiers remain and allow re-identification, and that "we'll mask it later" becomes "we didn't." Synthetic data avoids all of this and is usually less work than a defensible masking pipeline.

What's the difference between encryption and tokenization?

Encryption is mathematically reversible with the key — the ciphertext is derived from the plaintext. A token has no mathematical relationship to the value; it's a lookup key into a separate vault. That difference matters for scope: encrypted card data is still card data under PCI DSS, while tokenized card data takes the storing system out of scope.

How do we delete personal data from backups?

Generally you can't, selectively, and regulators broadly accept this. The workable approaches: define a backup retention window short enough that deleted data ages out (30–90 days), document that deletion propagates as backups expire, ensure restored backups get deletion requests reapplied, and use crypto-shredding — encrypt per-user with a per-user key and destroy the key, rendering the backup copy unreadable. See Data Retention & Deletion.

Does encryption exempt us from breach notification?

Often it reduces the obligation substantially — many regimes have a safe harbor for data that was encrypted with keys the attacker didn't obtain. The qualification matters: if the attacker got the application's credentials and the application can decrypt, the data was effectively in the clear. At-rest encryption on a database an attacker queried through the application provides no protection and no safe harbor.

How do we handle personal data in free-text fields?

Assume it contains everything. Support notes, feedback forms, and description fields accumulate names, phone numbers, health information, and occasionally card numbers. Options: classify the whole field at your highest sensitivity level, run detection to flag and redact patterns, restrict access more tightly than structured fields, and set a shorter retention period. Removing free-text fields where a structured alternative works is the cleanest fix.

Related Topics

References