AWS Step Functions: Serverless Workflow Orchestration

Q: Orchestration vs choreography — what's the difference?

Orchestration (Step Functions) has a central coordinator that explicitly directs each step — easy to reason about and observe, but the coordinator knows about every service. Choreography (EventBridge, events) has services react independently to events — loosely coupled and scalable, but the overall flow is emergent and harder to trace. Most mature systems mix both: choreograph across bounded contexts, orchestrate within a complex process.

Q: How do I model a human-approval step?

Use a Task with the .waitForTaskToken integration pattern. The workflow pauses and hands you a token (e.g. via an email or SQS message); when the human acts, your code calls SendTaskSuccess/SendTaskFailure with that token and the workflow resumes. Standard workflows can wait up to a year, which is what makes long approval processes feasible.

AWS Step Functions

AWS Step Functions is a serverless orchestration service — it coordinates multiple AWS services and functions into a workflow defined as a state machine. Instead of chaining Lambda functions with brittle glue code and hand-rolled retry logic, you describe the steps, the transitions, the error handling, and the parallelism declaratively, and Step Functions runs it, tracks state, and gives you a visual execution history.

It solves the problem every non-trivial serverless system eventually hits: a business process is more than one function. "Process an order" might mean validate → charge payment → reserve inventory → notify shipping → email the customer, with retries on transient failures, compensation on payment errors, and a branch for out-of-stock items. Encoding that in application code produces tangled, hard-to-observe systems. Step Functions makes the workflow itself a first-class, inspectable artifact — the orchestration side of coordinating distributed work.

TL;DR

Step Functions runs workflows as state machines — you define states (steps), transitions, retries, and error handling in JSON/YAML (Amazon States Language), and it executes and tracks them.
Two flavors: Standard (long-running up to 1 year, exactly-once, full audit history) and Express (high-volume, short-lived, at-least-once, cheaper).
Built-in retries, catch/fallback, timeouts, parallelism (Parallel), and iteration (Map) — resilience you'd otherwise hand-code.
Direct SDK integrations call ~200 AWS services without a Lambda wrapper; .waitForTaskToken enables human-approval and long-running callbacks.
Every execution is visual and auditable — you see exactly which state each run reached and why it failed.
It's orchestration (a central brain directing steps), the counterpart to choreography (services reacting to events via EventBridge); real systems use both.

Core Concepts

State Machines and the States Language

A workflow is a state machine: a set of states connected by transitions. You author it in Amazon States Language (ASL) — JSON describing each state's type and behavior:

Common state types: Task (do work — invoke Lambda, call a service), Choice (branch on input), Parallel (run branches concurrently), Map (iterate over an array, fanning out), Wait (pause), Pass, Succeed, and Fail. State passes JSON input/output between steps, so the workflow threads data through the pipeline.

Standard vs Express Workflows

This is the first architectural decision:

Use Standard for durable, auditable business workflows (order processing, approvals, data pipelines). Use Express for high-volume, short event processing (per-request microservice orchestration, Kinesis/ingestion handlers) where you'd otherwise pay a fortune in state transitions.

Error Handling, Retries, and Compensation

The biggest reason to adopt Step Functions is that resilience is declarative. Each Task supports Retry (with error matching, max attempts, backoff, jitter) and Catch (route specific errors to a fallback state). This lets you implement the saga pattern — on failure partway through a multi-step process, transition to compensating actions (refund the payment, release the inventory) rather than leaving the system inconsistent. Doing this correctly in raw Lambda glue is error-prone; here it's part of the workflow definition.

Service Integrations and Callbacks

Step Functions integrates directly with ~200 AWS services via SDK integrations — a Task can call DynamoDB, SQS, SNS, ECS, Bedrock, etc. without a Lambda in between, cutting cost and cold-start latency. For steps that wait on something external, the .waitForTaskToken pattern pauses the workflow and resumes when your code returns a token — the standard way to model human approval (send an email, wait for a click) or long-running third-party jobs inside a workflow.

Common Mistakes

Putting Business Logic in the State Machine

ASL can do choices and transformations, but it's a coordination language, not a place for complex logic. Cramming heavy conditionals, string manipulation, and calculations into Choice states and intrinsic functions makes workflows unreadable. Keep meaningful logic in Lambda (or the target service) and let Step Functions handle orchestration — the sequencing, retries, and branching.

Choosing Standard When Express Fits (or Vice Versa)

Running a high-volume, per-request orchestration as Standard racks up per-state-transition costs fast; running a long-lived approval workflow as Express breaks (5-minute cap, no exactly-once, no visual history). Match the workflow type to duration, volume, and auditability needs up front — switching later means re-testing semantics.

Ignoring Payload Size Limits

State input/output has a size limit (256 KB). Passing large blobs (files, big result sets) directly between states hits this ceiling. Store the payload in S3 (or DynamoDB) and pass a reference (key/ID) through the workflow instead — the classic "claim check" pattern.

Reaching for It When an Event Would Do

Not every multi-service flow needs central orchestration. If services simply react to things happening, choreography via EventBridge/SNS is looser-coupled and simpler. Use Step Functions when you genuinely need a coordinated, stateful process with ordering, branching, retries, and visibility — not as a default for any two connected functions.

Unbounded Map Concurrency

The Map state fans out over an array, and by default runs items with high concurrency — which can overwhelm downstream services or hit account limits. Set MaxConcurrency (or use Distributed Map for very large datasets with controlled batching) so a big input array doesn't stampede your databases or throttle other workloads.

FAQ

Step Functions vs a Lambda that calls other Lambdas?

A "driver" Lambda that invokes others works for trivial cases but you end up re-implementing retries, timeouts, error branching, and state tracking by hand — and you get no visibility into where a run failed. Step Functions provides all of that declaratively, plus a visual execution history. Once a flow has more than a couple of steps or needs real error handling, the state machine is far more maintainable.

Orchestration vs choreography — what's the difference?

Orchestration (Step Functions) has a central coordinator that explicitly directs each step — easy to reason about and observe, but the coordinator knows about every service. Choreography (EventBridge, events) has services react independently to events — loosely coupled and scalable, but the overall flow is emergent and harder to trace. Most mature systems mix both: choreograph across bounded contexts, orchestrate within a complex process.

How do I model a human-approval step?

Use a Task with the .waitForTaskToken integration pattern. The workflow pauses and hands you a token (e.g. via an email or SQS message); when the human acts, your code calls SendTaskSuccess/SendTaskFailure with that token and the workflow resumes. Standard workflows can wait up to a year, which is what makes long approval processes feasible.

Is Express cheaper?

For high-volume, short-lived workflows, yes — Standard is billed per state transition, which gets expensive at scale, while Express is billed by number of executions and duration/memory. But Express trades away exactly-once semantics and the visual audit history. Use Express for throughput, Standard for durability and auditability.

Can it replace a Kafka/Kinesis pipeline?

No — those are streaming logs for high-throughput, ordered, replayable records. Step Functions orchestrates discrete workflow executions. You might trigger an Express workflow per record from a Kinesis stream, but the stream itself is the pipe; Step Functions coordinates what happens to each item.