Background Jobs & Workers
Background jobs move slow or unreliable work out of the request/response cycle and onto workers that process it asynchronously. The user gets an instant response while emails, image processing, and report generation happen behind the scenes.
The shift is from "do it now, make the user wait" to "accept it now, do it reliably soon." That reliability — retries, idempotency, dead-letter handling — is what separates a robust job system from a fragile one.
TL;DR
- Offload slow or flaky operations (email, media, reports, third-party calls) to workers.
- Use a message queue for reliable delivery and retries.
- Design jobs to be idempotent so retries are safe.
- Keep payloads small and monitor queue depth and failures.
Quick Example
A producer enqueues work and returns immediately; a worker processes it out of band (BullMQ):
Core Concepts
When to use a background job
Email and notifications, image/video processing, report generation, data import/export, scheduled tasks, and slow third-party API calls — anything that would otherwise make a request hang or fail.
Queue patterns
- Work queue — multiple workers share the load (the common case).
- Pub/sub — one event fans out to many subscribers.
- Priority queue — urgent jobs jump ahead.
- Delayed / scheduled — run later or on a cron.
Job design principles
- Idempotent — safe to run twice (queues are usually at-least-once).
- Small and focused — one responsibility per job.
- Serializable payload — pass IDs and primitives (JSON), not live objects.
- Bounded — set timeouts; route repeated failures to a dead-letter queue.
See Message Queues for delivery semantics and DLQs.
Tools
Best Practices
Make jobs idempotent
Key work on a business ID and check before acting, so a redelivered job doesn't double-send or double-charge:
Pass references, not payloads
Enqueue an ID and let the worker fetch the data — large payloads bloat the queue and can go stale.
Monitor the queue
Track queue depth, job age, failure rate, and DLQ count, and alert when they spike. A silently backed-up queue is an outage in waiting. See Logging.
Common Mistakes
Non-idempotent jobs
Embedding huge payloads
FAQ
When should work become a background job?
Whenever it's slow, unreliable, or doesn't need to finish before responding to the user — sending email, processing media, generating reports, or calling flaky third-party APIs. If the user doesn't need the result immediately, offload it.
How do I make a job safe to retry?
Make it idempotent: key on a business ID, check whether the work was already done, and use conditional writes. Since most queues deliver at-least-once, assume any job can run more than once.
Background jobs or a message queue — what's the difference?
Background jobs are the pattern (do work off the request path); a message queue is the infrastructure that delivers jobs reliably to workers. Job libraries like BullMQ or Celery are built on a queue/broker (often Redis).
How do I run scheduled or recurring jobs?
Use your job library's scheduler (cron-style triggers, delayed jobs, or run_at timestamps). The scheduler enqueues a normal job at the right time, which a worker then processes.
Related Topics
- Message Queues — The delivery layer underneath
- Redis — A common queue backend
- Microservices — Distributed job processing
- Rate Limiting — Throttling worker throughput