AWS Kinesis: Real-Time Data Streaming

Q: Data Streams or Firehose?

Use Firehose when you just want to reliably land a stream into S3/Redshift/OpenSearch with minimal effort and near-real-time (buffered) delivery — no code. Use Data Streams when you need custom real-time processing, multiple independent consumers, strict ordering, low latency, or replay. They also compose: a Data Stream feeding a Firehose for archival while other consumers process live.

Q: How does ordering work?

Ordering is per shard, not per stream. Records with the same partition key go to the same shard and are read in order; records across different shards have no global order. If you need per-entity ordering (e.g. all events for one user in order), make the entity ID the partition key so they share a shard.

Q: How do I scale throughput?

Add shards (each adds ~1 MB/s in, 2 MB/s out) in provisioned mode, or switch to on-demand to auto-scale. For read fan-out to many consumers without contending for the shared 2 MB/s read limit, use Enhanced Fan-Out, which gives each consumer a dedicated pipe. Watch throughput and iterator age in CloudWatch to know when to scale.

AWS Kinesis

Amazon Kinesis is AWS's managed real-time streaming platform — it ingests high-volume, continuous data (logs, clickstreams, IoT telemetry, application events) as an ordered, durable, replayable stream that many consumers can read independently. It's the AWS-native answer to Apache Kafka: a distributed commit log for data in motion, without running your own broker cluster.

The core problem it solves is decoupling producers from consumers at high throughput while preserving order and replay. A message queue like SQS deletes a message once it's processed and doesn't preserve global order; a stream retains records for a window (24 hours to 365 days) so multiple, independent consumers can each read the whole thing at their own pace, replay from any point, and process in order per key. That makes Kinesis the backbone for analytics pipelines, real-time dashboards, event sourcing, and fan-out to many downstream systems — the streaming layer of data engineering on AWS.

TL;DR

Kinesis is managed streaming — an ordered, durable, replayable log for high-volume real-time data. Think AWS-native Kafka.
Kinesis Data Streams is the core: records are partitioned into shards by partition key; order is guaranteed within a shard.
Kinesis Data Firehose is the easy button — a fully managed pipe that buffers and delivers streaming data to S3, Redshift, OpenSearch, or Splunk with no consumer code.
Multiple consumers read independently and checkpoint their position; replay is possible within the retention window.
Streams vs SQS: streams retain data for many readers and preserve order; queues delete-on-consume for a single work pool. Different tools.
Capacity is provisioned (shards) or on-demand (auto-scaling); shard math and hot partition keys are the classic operational traps.

Core Concepts

The Kinesis Family

"Kinesis" is several services; know which one you mean:

Most "should I use Kinesis?" decisions are between Data Streams (you need custom processing, multiple consumers, replay, ordering) and Firehose (you just want streaming data landed in a store with zero code).

Shards, Partition Keys, and Ordering

A Data Stream is divided into shards — the unit of both capacity and ordering. Each shard handles a fixed throughput (roughly 1 MB/s or 1,000 records/s in, 2 MB/s out). Every record carries a partition key; Kinesis hashes it to pick a shard, and order is guaranteed within a shard (not across the stream):

Choosing partition keys well is critical: keys that concentrate traffic create a hot shard that throttles while others sit idle. Use a key with good cardinality and even distribution (e.g. a user or device ID), not a constant or low-cardinality value.

Consumers and Checkpointing

Unlike a queue, a stream doesn't delete records when read — consumers track their own position. With the classic shared throughput model, consumers poll (via the Kinesis Client Library, which handles shard assignment and checkpointing to DynamoDB); with Enhanced Fan-Out, each consumer gets its own dedicated 2 MB/s per shard and records are pushed with lower latency. Lambda can consume a stream directly (polling shards in batches). Because position is a checkpoint, a consumer that fails can resume where it left off, and you can replay from an earlier point within the retention window — the superpower queues don't have.

Firehose: Streaming Without Code

Data Firehose is the managed delivery path: point producers at a Firehose stream and it buffers records (by size or time), optionally transforms them (via Lambda), compresses, and delivers to S3, Redshift, OpenSearch, or third-party sinks — no consumers, no shard management, no checkpointing. It's not truly sub-second (it buffers), but for "get my event/log stream into the data lake" it's the least-effort option by far.

Common Mistakes

Using Kinesis Where SQS Would Do

If you just need to hand work to a pool of workers that each process a message once and move on, SQS is simpler and cheaper — no shards, no checkpointing, automatic per-message retry and DLQ. Reach for Kinesis only when you genuinely need ordering, multiple independent consumers, replay, or high sustained throughput. Picking a stream for a plain work queue adds real operational complexity for nothing.

Poor Partition Key Choice (Hot Shards)

A low-cardinality or skewed partition key funnels most records into one shard, which throttles (ProvisionedThroughputExceeded) while other shards idle. Choose a high-cardinality, evenly distributed key. If traffic is inherently skewed, consider on-demand capacity or a composite key to spread load. Hot shards are the single most common Kinesis performance problem.

Under- or Over-Provisioning Shards

With provisioned mode you pay per shard and must resize (split/merge) as traffic changes — too few throttles, too many wastes money. Teams either under-provision and drop data or over-provision and overpay. If your traffic is spiky or hard to predict, on-demand mode auto-scales and removes the shard-math guesswork (at a higher per-record price). Monitor throughput in CloudWatch.

Forgetting the Retention Window

Records expire after the retention period (default 24 hours, extendable to 365 days). If a consumer is down longer than retention, its unread data is gone — no replay possible. Set retention to cover your worst-case consumer downtime and reprocessing needs, and alarm on iterator age (how far behind a consumer is) so you catch lag before data ages out.

Ignoring Consumer Lag (Iterator Age)

A slow or stuck consumer falls behind; the GetRecords.IteratorAgeMilliseconds metric shows how stale the records it's reading are. Rising iterator age means the consumer can't keep up and risks losing data at retention. Scale consumers (more shards / Enhanced Fan-Out / more Lambda concurrency) or speed up processing, and alarm on iterator age in CloudWatch.

FAQ

Kinesis vs Kafka?

Both are distributed, ordered, replayable streaming logs, and the concepts map closely (Kinesis shard ≈ Kafka partition). Kinesis is fully managed by AWS — no clusters to run, tight AWS integration, simpler ops — but it's AWS-only and has hard per-shard limits. Kafka (self-managed or via Amazon MSK) is more flexible, portable across clouds, has a huge ecosystem, and often cheaper at very high scale — at the cost of more operational work. Choose Kinesis for low-ops AWS-native streaming, Kafka/MSK for portability, ecosystem, or extreme scale.

Kinesis vs SQS/SNS?

SQS is a work queue: one message, consumed once, then deleted; no global ordering (except FIFO within a group). SNS is pub/sub fan-out. Kinesis retains records so many consumers can each read the whole stream, preserves order per shard, and allows replay. Use queues for task distribution, streams for real-time data pipelines with multiple/replayable consumers.

Data Streams or Firehose?

Use Firehose when you just want to reliably land a stream into S3/Redshift/OpenSearch with minimal effort and near-real-time (buffered) delivery — no code. Use Data Streams when you need custom real-time processing, multiple independent consumers, strict ordering, low latency, or replay. They also compose: a Data Stream feeding a Firehose for archival while other consumers process live.

How does ordering work?

Ordering is per shard, not per stream. Records with the same partition key go to the same shard and are read in order; records across different shards have no global order. If you need per-entity ordering (e.g. all events for one user in order), make the entity ID the partition key so they share a shard.

How do I scale throughput?

Add shards (each adds ~1 MB/s in, 2 MB/s out) in provisioned mode, or switch to on-demand to auto-scale. For read fan-out to many consumers without contending for the shared 2 MB/s read limit, use Enhanced Fan-Out, which gives each consumer a dedicated pipe. Watch throughput and iterator age in CloudWatch to know when to scale.