Security Regression Tests: Keep Vulnerabilities Fixed

Security Regression Test Suites

Fixed vulnerabilities have a way of coming back. A refactor moves the authorization check, a new endpoint copies an old pattern, a library upgrade changes how input is parsed, or a second code path to the same data appears without the fix. A security regression test makes each fix permanent: it encodes the attack that used to work as an automated test that must keep failing — for the attacker — on every build.

The suite grows from real findings — penetration tests, bug bounty reports, incidents, code review — and from the abuse cases identified in threat models. Over time it becomes an executable record of everything the team has learned about how its system can be attacked, and a cheap safety net that runs in minutes.

TL;DR

Every confirmed vulnerability gets a negative test before the fix is merged: the test fails on the vulnerable code and passes on the fixed code.
Test at the trust boundary — the HTTP endpoint, message consumer, or upload handler — not only the internal helper that was patched.
Cover bypass variants: encodings, alternative IP formats, case changes, redirects, other HTTP methods, and other paths to the same data.
Tag each test with the finding it protects against, so nobody deletes it as redundant.
Run the suite in CI on every change, alongside a lightweight DAST baseline scan.
Keep tests fast and deterministic; a slow, flaky security suite gets skipped.

Quick Example

Regression tests for two past findings — an SSRF in webhook URLs and a stored XSS in project names — written as ordinary pytest integration tests.

The parameter list encodes what the team learned while fixing the SSRF: the first patch blocked 169.254.169.254 but not the integer and IPv6 forms.

Core Concepts

What Belongs in the Suite

Every confirmed vulnerability, from any source: penetration tests, bug bounties, incidents, scanners, and code review.
Abuse cases from threat models that the design explicitly defends against.
Security requirements that must hold permanently, such as "cross-tenant requests return 404" or "session IDs rotate at sign-in".
Near misses — weaknesses found before they were exploitable.

Test at the Boundary

A vulnerability is fixed for users only if the externally reachable path is safe. If the fix was in a URL-validation helper, test the endpoint that accepts the URL, because another endpoint might not use the helper, or the helper might be bypassed after a refactor. Unit tests on the helper are useful additions, not substitutes.

Variants Matter

Attackers rarely stop at the payload in the original report. A regression test should cover the family of inputs the fix is meant to block:

Traceability

Tag each test with the finding ID or a short description (as in the security marker above). Tags let you list which findings have coverage, report it to auditors, and stop well-meaning cleanup from deleting a test that "looks redundant".

Organizing the Suite

Where the Tests Live

Authorization deserves its own matrix-driven tests — see Authorization Testing Across Tenant Boundaries.

Running in CI

Run the security-tagged tests with every pull request, and the DAST baseline against a deployed preview or CI environment. Failures should block merging like any other failing test; security regressions are bugs.

Best Practices

Write the Test Before the Fix

Reproduce the vulnerability as a failing test, then fix it and watch the test pass. This proves the test actually detects the issue and that the fix addresses it.

Assert on Effects, Not Just Status Codes

A 422 is not enough if the server already made the outbound request. Where possible, assert that the side effect did not happen: no outbound connection, no row changed, no file written, no data in the response body.

Keep Payloads Benign but Realistic

Use payloads that demonstrate the issue without causing harm — alert(1) rather than credential theft, internal addresses that point nowhere in CI. Store them in the repository like any other test data.

Review the Suite After Incidents

After every security incident or significant finding, ask which test would have caught it and add it. The suite should grow with experience.

Common Mistakes

Testing Only the Exact Reported Payload

Testing the Helper but Not the Endpoint

Tests That Pass Because the Feature Is Off

FAQ

What is a security regression test?

An automated test that reproduces a previously fixed vulnerability and asserts it stays fixed. It fails if a later change reintroduces the weakness, catching the regression before it reaches production.

How is this different from penetration testing?

Penetration testing is a periodic, expert-led search for new vulnerabilities. Regression tests automatically and continuously confirm that known vulnerabilities remain fixed. Findings from penetration tests are one of the main sources of new regression tests.

Which bugs deserve a regression test?

Every confirmed vulnerability, plus security requirements and abuse cases the design must always defend against. The cost of a test is small compared with rediscovering the same vulnerability in production.

Is it safe to keep exploit payloads in the repository?

Yes, when they are benign demonstrations against your own test environment, such as alert(1) or requests to unroutable internal addresses. Avoid real credentials, destructive payloads, or targets outside your control.

How do we stop the suite becoming slow?

Test at the API layer rather than through the browser where possible, parameterize variants into a single fast test, run the full DAST scan nightly while keeping a quick baseline in pull requests, and remove duplication — but never delete a test without confirming its finding is still covered.