Secret Rotation & Credential Leak Response

Secret Rotation and Leak Response

An API key pushed to a public repository is typically found by automated scanners within minutes. A database password printed in CI logs is visible to everyone with access to the pipeline. A cloud access key baked into a container image travels wherever the image goes. Leaks are inevitable at any scale; what matters is how quickly they are detected, how decisively the credential is revoked, and whether every service can switch to a new secret without an outage.

Teams that cannot rotate secrets quickly face a terrible choice during an incident: leave a compromised credential active, or break production by revoking it. Rotation designed and practiced in advance removes that choice.

TL;DR

Secrets leak through git history, logs, CI output, container images, client bundles, chat, and tickets.
Detect with secret scanning (push protection, repository and image scanning), provider alerts, and anomaly monitoring.
On a leak: revoke or rotate immediately, then investigate usage, contain, clean up, and fix the root cause.
Deleting the secret from git is not remediation — assume it was copied.
Rotate without downtime using two valid credentials during the switch, versioned secrets, and reloadable configuration.
Prefer short-lived, dynamic credentials and workload identity, and drill rotations regularly.

Quick Example

First minutes of responding to a leaked AWS access key (using AWS's documentation example key ID):

Zero-downtime rotation for a credential used by many running services:

Core Concepts

How Secrets Leak

Detection

Push protection blocks commits containing known secret patterns before they reach the remote.
Repository, image, and log scanning with tools such as GitHub secret scanning, gitleaks, and TruffleHog finds secrets already present.
Provider partner programs notify you — and sometimes revoke automatically — when your keys appear in public code.
Canary tokens: decoy credentials that alert when used.
Usage anomalies: API calls from unusual locations, times, or volumes.

Leak Response Steps

Revoke or rotate the credential immediately. Assume compromise; scanners and attackers copy secrets within minutes.
Investigate usage through provider audit logs (CloudTrail, GitHub audit logs, database logs): what was accessed, from where, and when.
Contain: revoke sessions, block attacker infrastructure, rotate related credentials the leaked one could reach.
Clean up: remove the secret from history (git filter-repo), purge caches, delete exposed artifacts and logs — after rotation, not instead of it.
Recover and fix: restore anything changed, close the root cause, and add prevention (scanning, short-lived credentials).
Notify as required by policy, contracts, or regulation. See Security Incident Response.

Rotation Without Downtime

Short-Lived and Dynamic Credentials

The best leaked secret is one that expires before it can be used. Workload identity (cloud IAM roles for services, OIDC federation for CI) eliminates long-lived keys entirely, and dynamic secrets from systems such as HashiCorp Vault issue unique database or cloud credentials per service with short leases, revoked automatically.

Rotation Drills

Rotate every important credential on a schedule — even without a leak — to prove the process works. Track for each secret: owner, where it is used, how to rotate it, blast radius, and the last successful rotation. A secret nobody knows how to rotate is a future outage.

Best Practices

Keep an Inventory of Secrets

Know every credential, its owner, its consumers, and its rotation procedure, ideally in the secret manager's metadata. See Secrets Management.

Never Put Secrets in Code, Images, or Client Bundles

Inject them at runtime from a secret manager, and scan builds to enforce it.

Scope Every Credential Narrowly

Least-privilege, per-service credentials limit what a leak exposes and make rotation affect fewer systems.

Log Usage, Not Values

Audit which identity used which credential and when, and redact secret values from all logs. See Audit Logging.

Common Mistakes

Deleting the Commit Instead of Rotating

Rotation That Requires a Coordinated Outage

Revoking Before Knowing the Consumers

FAQ

What should I do first if a secret leaks?

Revoke or rotate it immediately, assuming it has been copied. Then investigate how it was used, contain any damage, clean it out of the places it leaked, and fix the cause.

Is removing a secret from git history enough?

No. Once pushed, especially publicly, a secret should be considered compromised. Rotation is the remediation; history cleanup only reduces further exposure.

How do I rotate a secret without downtime?

Make a new credential valid alongside the old one, update the secret store, let services pick up the new value, confirm the old credential is no longer used, then revoke it.

How often should secrets be rotated?

Regularly enough that the process stays practiced — often every 30 to 90 days for long-lived credentials, depending on policy — and immediately after any suspected exposure. Short-lived credentials remove most of the need.

What are dynamic secrets?

Credentials generated on demand for each consumer with a short lifetime, then revoked automatically — for example, per-service database users issued by a secrets manager — so leaked values expire quickly.