Audit Logging
An audit log answers one question: who did what, to which data, and when? It's the evidence that a control was operating, the material a breach investigation depends on, and — increasingly — a legal requirement. SOC 2 asks for it, GDPR breach assessment is impossible without it, HIPAA mandates it for access to health records, and PCI DSS specifies exactly what must be captured.
It's also routinely conflated with application logging, which is a mistake with real consequences. Application logs are for debugging: verbose, unstructured, ephemeral, and written by developers on demand. Audit logs are for evidence: structured, complete, immutable, and retained for years. A logger.info("user updated their email") buried among a million debug lines, rotated away after 14 days, is not an audit trail — and discovering that during an incident is a bad time to find out.
The distinguishing property is tamper-evidence. An audit log an administrator can quietly edit proves nothing, which is why append-only storage and hash chaining matter more here than the volume of what you capture.
TL;DR
- Audit logs are evidence, not diagnostics. Keep them separate from application logs.
- Record who, what, when, where, and outcome — including failed attempts.
- Append-only and tamper-evident. Hash chaining makes modification detectable.
- Log reads of sensitive data, not just writes. Access is the thing breaches consist of.
- Never put the sensitive values in the audit log — record identifiers and field names, not payloads.
- Include a correlation ID so an action can be traced across services.
- Retention is years, not weeks — often 1–7 depending on the regime.
- Design the queries you'll need during an incident before the incident.
Quick Example
An audit event schema and a hash-chained writer:
Two details matter disproportionately. fields=["email"] records what changed without duplicating the personal data into a second, long-retained store. And the denial is logged — a sequence of denied attempts is exactly the signal an intrusion produces, and systems that only log successes are blind to it.
Core Concepts
Audit log vs. application log
The failure mode is treating one as the other. Audit events buried in an application log stream are unfindable, get rotated away, and can be edited by anyone with log-store access. Debug messages written to the audit store bloat it and make the evidence harder to read.
Write audit events through a dedicated path to a dedicated store.
What to log
Logging reads is the requirement teams most often skip and auditors most often ask about. A breach is someone reading data they shouldn't; without read logging, you cannot tell what was taken, and "we don't know" is the answer no regulator accepts.
Impersonation deserves particular care. Support tools that let an agent act as a user are enormously useful and are the highest-risk feature in most products — always record both the real actor and the impersonated subject.
Tamper evidence
Hash chaining makes modification detectable, not impossible. Strengthen it by periodically publishing the chain head somewhere the operator can't alter — a separate account, a signed external timestamp, or a customer-visible attestation.
Storage-level immutability is the complementary control:
The principle to aim for: the people whose actions are audited should not control the audit store. An administrator who can delete their own audit trail has an audit trail that proves nothing.
Storage
Partition by time — it makes retention a partition drop instead of a mass delete, and it keeps recent-window queries fast. See PostgreSQL Partitioning.
At higher volume, the common architecture is append to a durable stream (Kafka, Kinesis) and materialize into both a queryable store and immutable archival storage. That also decouples audit writes from the request path.
Retention
Note the tension: GDPR's minimization principle applies to audit logs too. An audit log full of personal data retained for seven years is itself a compliance problem, which is the practical argument for logging identifiers and field names rather than values.
Querying under pressure
Design these before you need them:
Writing and testing these in advance is the difference between a two-hour investigation and a two-day one. Put them in the runbook.
Best Practices
Emit audit events at the domain layer, not the HTTP layer
An audit event should describe a business action — subscription.cancelled — not POST /api/v2/subs/123. Domain-level events survive API refactors, are meaningful to auditors, and capture actions that arrive through multiple paths (API, admin tool, background job).
Log denials as prominently as successes
A sequence of authorization failures is the clearest intrusion signal available, and systems that only record successful actions are blind to reconnaissance entirely.
Never write sensitive values into the audit log
Record fields_changed: ["email", "phone"], not the old and new values. The audit log is long-retained, widely accessible for investigation, and would otherwise become a second copy of the personal data you're trying to protect.
Make the audit write path non-optional
Audit through a decorator, middleware, or database trigger rather than a call developers must remember. A code path that skips the audit event is a gap you'll discover during an investigation.
Separate the audit store from the operational one
Different database, different account, different credentials — ideally where the application's own administrators have insert-only access. Blast-radius separation is what makes the log credible.
Include a correlation ID
One user action often spans several services. A request ID threaded through every event lets you reconstruct the whole action rather than seeing fragments. See Distributed Tracing.
Alert on audit patterns, not just collect them
Bulk exports, off-hours admin access, permission escalations, and denial spikes should page someone. A log nobody watches provides forensics after the fact and no detection during. See Alerting.
Verify the chain regularly
A scheduled job that verifies the hash chain and alerts on a break. An integrity mechanism nobody checks is one you'll discover was broken during the investigation that needed it.
Common Mistakes
Audit events mixed into application logs
Not logging reads
Sensitive values in the log
A mutable audit store
Retention shorter than the requirement
Audit failures breaking the request
That last one is a genuine design decision rather than a bug. For high-sensitivity systems, failing the action when it cannot be audited is the correct behavior; for most systems, a durable queue with alerting is the right balance. What's not acceptable is silently proceeding without a record.
FAQ
Where should audit logs live?
Somewhere the audited parties don't control. A separate database, a separate cloud account, or a managed service (AWS CloudTrail for infrastructure, a dedicated audit service for application events). For archival, immutable object storage with a legal-hold or compliance-mode lock. Keeping them in the same database with the same credentials as the application undermines the point.
How much does this cost at scale?
Less than expected, because audit volume is far lower than application log volume — you're recording business actions, not every function call. Cost control comes from partitioning by time, tiering older partitions to cheap object storage, and logging identifiers rather than payloads. A system doing millions of requests a day typically generates thousands of audit events.
Do we need to log every read?
Every read of sensitive data, yes — that's what HIPAA and PCI DSS require and what breach investigation depends on. Reads of public or non-sensitive data don't warrant it. For high-volume read paths, per-query logging (with the filter and the row count) rather than per-record keeps the volume manageable while preserving the signal.
Can we use application logs as an audit trail?
Only if they meet the audit requirements: structured, complete, immutable, appropriately retained, and separated from debug noise. That's usually a bigger change than adding a dedicated path. The reliable test: can you produce a complete, verifiable list of everyone who accessed a specific record in the past year, in under an hour?
Is blockchain useful for audit logs?
Almost never. A hash chain gives you the tamper-evidence, and periodically publishing the chain head to an external system gives you the third-party attestation. A blockchain adds cost, latency, and operational complexity for a property you already have — plus the immutability becomes a liability when an audit entry needs to be redacted for a legal reason. See Blockchain Fundamentals for when distributed consensus is actually warranted.
What about database-level auditing?
pgaudit for PostgreSQL, audit plugins for MySQL, and native features in SQL Server capture every statement at the database layer. That's excellent complementary coverage — it catches direct database access that bypasses the application entirely — and it's a poor substitute for application-level auditing, because it records SQL rather than business intent. Use both: database auditing for the "someone connected directly" case, application auditing for everything else.
Related Topics
- SOC 2 & Security Compliance — Where audit logs are evidence
- GDPR for Engineers — Breach assessment depends on this
- PII & Sensitive Data Handling — What not to put in the log
- Data Retention & Deletion — Retention applies to audit logs too
- Logging — Application logging, kept separate
- Log Aggregation — Centralizing and searching at scale
- Alerting — Detection built on audit patterns
- Security — The broader control set
- PostgreSQL Partitioning — Time-partitioned audit tables