Graceful Shutdown, Health Probes & Service Lifecycle

Backend Service Lifecycle and Graceful Shutdown

Backend processes are stopped far more often than most developers think. Every deploy, autoscaling decision, node upgrade, and spot-instance reclaim ends some processes and starts others — often many times a day. A service that exits abruptly drops the requests it was handling, abandons half-finished jobs, and leaves clients with a burst of 502 errors on every release. A service that reports itself ready before it can serve does the same thing on the way up.

Handling the lifecycle well is unglamorous and mechanical: start only when ready, tell the platform the truth about health, and on shutdown stop taking new work, finish what is in flight, release resources, and exit before the deadline. Done properly, deploys become invisible to users.

TL;DR

Treat startup and shutdown as features; they run on every deploy and scale event.
Use startup, liveness, and readiness probes for different questions, and keep dependency checks out of liveness.
On SIGTERM: fail readiness, wait for traffic to stop arriving, stop accepting work, drain in-flight requests, close resources, and exit.
Make sure your process actually receives signals — use exec-form commands or an init process in containers.
Align timeouts: drain time below the orchestrator's grace period; app keep-alive timeout above the load balancer's idle timeout.
Make background work idempotent and resumable, because hard kills still happen.

Quick Example

A Node.js HTTP server that drains cleanly when Kubernetes, Docker, or systemd sends SIGTERM:

The order matters. Stopping the listener immediately would refuse requests that load balancers are still sending; never stopping would let the platform kill the process mid-request when the grace period ends.

Core Concepts

Lifecycle Phases

Health Probes

A liveness probe that checks the database is a classic way to turn a short database blip into a fleet-wide restart storm: every replica fails liveness at once, restarts, and reconnects simultaneously.

Signals

In a container, the main process runs as PID 1, which receives no default signal handling, and shell-form commands put /bin/sh in that position — and the shell does not forward SIGTERM to your application. Use exec-form CMD ["node", "server.js"], exec at the end of entrypoint scripts, or a small init such as tini. See Linux Processes & Signals.

The Graceful Shutdown Sequence

Receive SIGTERM and log it.
Fail readiness and wait a few seconds, so load balancers and service discovery stop routing new requests here.
Stop accepting new work: close the listening socket and stop pulling from queues.
Drain in-flight requests. Ask long-lived connections — WebSockets, server-sent events, gRPC streams — to reconnect elsewhere.
Finish or return background work: complete the current job within the deadline, or release it so another worker picks it up.
Flush and close: send buffered logs, traces, and metrics; close database pools and message clients.
Exit with code 0 before the deadline. Anything still running is killed with SIGKILL.

The same sequence in Go, where http.Server.Shutdown waits for active requests:

Most frameworks provide the draining step: Go's Shutdown, Node's server.close(), Gunicorn's --graceful-timeout, Uvicorn's --timeout-graceful-shutdown, and Spring Boot's server.shutdown=graceful. Your job is the surrounding sequence and the timing.

Kubernetes Termination Timing

When a pod is deleted, Kubernetes runs any preStop hook, then sends SIGTERM, while removing the pod from service endpoints at the same time. Endpoint changes take a few seconds to reach every load balancer and proxy, so a pod can receive new requests after SIGTERM. Either delay in the application, as in the examples above, or sleep in a preStop hook:

The native sleep action needs a recent Kubernetes version; older clusters use an exec hook running sleep, which requires the binary in the image. The preStop time counts against terminationGracePeriodSeconds.

Starting Up Well

Load and validate configuration first, and exit with a clear error if anything is missing — see Configuration and Environment Management.
Connect to dependencies with retries and backoff, so a slow database does not cause a crash loop, but give up eventually with a clear log message.
Warm up caches, connection pools, or JIT-compiled paths before reporting ready if the first requests would otherwise be slow.
Do not run schema migrations on every replica's startup: replicas race each other, and a failing migration blocks the whole rollout. Run migrations as a separate deploy step — see Database Migrations.
Report ready only when you can serve. A fast readiness signal that lies is worse than a slower honest one.

Background Workers and Long-Lived Connections

Queue consumers stop fetching new messages on SIGTERM, finish the current one if it fits within the deadline, and otherwise let it return to the queue — by negative acknowledgement or by letting its visibility timeout expire. Because a worker can always be killed mid-job, handlers must be idempotent, and long jobs should checkpoint progress or be split into smaller steps. See Background Jobs.

Long-lived connections need their own goodbye: send a WebSocket close frame or an HTTP/2 GOAWAY, stagger reconnections so thousands of clients do not reconnect in the same second, and make clients resume from where they left off.

Best Practices

Test Shutdown Under Load

Run a load test, roll the deployment or send SIGTERM to an instance, and assert zero failed requests. Shutdown bugs hide until traffic is real, and this test finds them in minutes.

Log Lifecycle Events

Log "starting", "ready", "draining", and "exited" with timestamps and durations. When a deploy causes errors, those lines show whether the process drained, timed out, or was killed.

Keep Startup Fast

Fast startup makes autoscaling responsive and rollbacks quick. Keep images small, defer rarely used initialization, and measure time-to-ready as a metric.

Assume the Hard Kill Will Happen

Out-of-memory kills, node failures, and exceeded deadlines bypass graceful shutdown entirely. Graceful shutdown reduces errors; idempotency and retries make the remaining ones harmless.

Common Mistakes

Shell-Form Commands That Swallow Signals

Dependency Checks in Liveness

Exiting Immediately on SIGTERM

A Grace Period Shorter Than the Drain

FAQ

What is the difference between liveness and readiness probes?

Liveness asks whether the process is so broken it needs restarting; failing it restarts the container. Readiness asks whether the instance should receive traffic right now; failing it removes the instance from load balancing without a restart. Draining instances and those still warming up should fail readiness, not liveness.

Why do we see 502 errors during every deployment?

Usually one of three causes: the process exits before load balancers stop routing to it, it does not receive SIGTERM at all and is killed, or the load balancer reuses a keep-alive connection that the application just closed. Fail readiness and wait before closing, check signal delivery in the container, and set the application's keep-alive timeout above the load balancer's idle timeout.

How long should the grace period be?

Long enough for the propagation delay, the longest normal request, and cleanup — commonly 30 to 60 seconds for web services. Workers running long jobs may need several minutes, or better, jobs that checkpoint so they can stop quickly.

Should readiness check the database?

Only cautiously. If every replica depends on the same database, failing readiness everywhere during a database outage removes every instance at once, turning fast error responses into connection failures. Many teams keep readiness focused on the instance itself and handle shared dependency failures with timeouts, circuit breakers, and alerts.

What happens if my application ignores SIGTERM?

The platform waits for the grace period, then sends SIGKILL, which cannot be handled. Every in-flight request and job fails, and every deploy takes the full grace period per instance.