Performance Testing
Performance testing answers a question functional tests can't: does the system hold up under load? It measures how an application behaves as traffic rises — its latency, throughput, error rate, and resource use — so you can validate it against requirements before real users find the breaking point.
The key to useful results is realism: test with production-like data, traffic patterns, and environments, and measure the percentiles that users actually feel (p95, p99) rather than averages — an average hides the slow tail where real pain lives.
TL;DR
- Define performance requirements upfront (latency, throughput, error rate).
- Test with realistic data and traffic patterns.
- Monitor during tests — CPU, memory, latency, connections.
- Test in production-like environments.
Quick Example
A k6 script ramps load and fails if latency or errors breach thresholds:
Core Concepts
Test types
- Load — expected traffic levels.
- Stress — beyond expected capacity, to find the breaking point.
- Spike — sudden bursts of traffic.
- Soak — sustained load over hours/days (surfaces leaks).
- Benchmark — comparing performance across code changes.
Key metrics
- Latency — response time, reported as p50/p95/p99, not just the average.
- Throughput — requests per second.
- Error rate — % of failed requests.
- Resource usage — CPU, memory, connections.
Requirements (example)
Tools
k6 (scripted, developer-friendly)
Artillery (YAML scenarios)
Benchmarking code changes (pytest-benchmark)
Realistic Traffic & CI Integration
Model real user mix rather than hammering one endpoint:
Gate CI on regressions:
Best Practices
- Set requirements first — you can't pass a test with no defined target.
- Measure percentiles (p95/p99), never just the mean.
- Use realistic data and traffic mixes — synthetic uniform load misleads.
- Test in a production-like environment — results from a laptop don't transfer.
- Monitor the system under test (CPU/memory/connections) to find the actual bottleneck.
Common Mistakes
Reporting averages instead of percentiles
Unrealistic test conditions
FAQ
What's the difference between load, stress, and soak testing?
Load testing checks behavior at expected traffic. Stress testing pushes beyond capacity to find where and how the system breaks. Spike testing throws sudden bursts to test elasticity. Soak (endurance) testing runs sustained load for hours to surface slow problems like memory leaks and connection exhaustion. Each targets a different failure mode — most teams need load + a periodic stress/soak run.
Why measure p95/p99 instead of average latency?
Averages hide the tail. A service can average 90ms while 1% of requests take 4 seconds — and that 1% is often your most active users, hitting the slowest paths. Percentiles (p95 = 95% of requests are at least this fast) describe the experience real users get at the edges, which is what SLAs and user satisfaction actually depend on.
Should performance tests run in CI?
Yes — at least a lightweight smoke load test that gates on key thresholds (e.g. p95 latency, error rate) to catch regressions before they ship. Full-scale load tests are expensive and better run on a schedule or before major releases in a production-like environment. The CI version trades thoroughness for fast feedback on obvious regressions.
How do I make load tests realistic?
Model actual user behavior: a weighted mix of journeys (most users browse, some search, few purchase), realistic think-time between actions, production-like data volumes, and warm caches where appropriate. Generate load from enough machines to actually stress the target, and run against an environment that mirrors production. Hammering a single endpoint from one box produces numbers that won't hold in reality.
Related Topics
- Monitoring — Observing systems under load
- Caching — A primary performance lever
- API Testing — Functional correctness first
- CI/CD — Automating regression gates
- High Availability — Designing for load