Google Cloud Pub/Sub: Global Messaging & Event Streaming

Q: Is Pub/Sub messaging or streaming?

Both — it's a pub/sub messaging system (fan-out to multiple subscribers, buffered delivery) and a streaming ingestion system (high-throughput event pipelines feeding BigQuery/Dataflow). This dual nature is distinctive: on GCP it serves the roles that AWS SNS (pub/sub) and Kinesis/Kafka (streaming) play, in one serverless service.

Q: Push or pull subscriptions?

Pull for high-throughput, consumer-controlled processing (workers fetch, batch, and ack at their own pace) — more control and efficiency at scale. Push for serverless and simple consumers — Pub/Sub POSTs messages to an HTTP endpoint or triggers a Cloud Function, so you don't run a polling consumer. Choose by consumer architecture; push is convenient for event-driven serverless, pull for dedicated high-volume workers.

Q: How does Pub/Sub compare to Kafka?

Pub/Sub is fully serverless — no partitions, brokers, or capacity to manage, auto-scaling to millions of messages/sec. Kafka offers stronger replay (retained log, replay by offset) and a richer streaming ecosystem but requires managing (or paying for managed) partitions and brokers. For GCP event-driven messaging and ingestion with zero ops, Pub/Sub; for replay-heavy streaming and Kafka's ecosystem, Kafka. Pub/Sub also has a Kafka-compatible offering.

Q: Does each consumer get every message?

Each subscription gets its own copy of every message on the topic — so multiple subscriptions mean multiple independent consumers each receive the full stream (fan-out). Within a single subscription, multiple consumer instances share the load (each message goes to one of them). So: fan-out across subscriptions, load-sharing within a subscription.

Q: How do I get exactly-once processing?

Enable exactly-once delivery on the subscription (eliminates duplicates within it, at some throughput cost), and/or design idempotent consumers (dedupe by message ID). Default is at-least-once (duplicates possible on redelivery), so for workloads where duplicates are unacceptable, use exactly-once delivery plus idempotency for defense in depth.

Google Cloud Pub/Sub

Google Cloud Pub/Sub is GCP's fully managed, globally-scalable messaging and event-streaming service — publishers send messages to topics, and subscriptions deliver them to consumers, decoupling the two so they don't communicate directly. It's the backbone of event-driven architectures, data pipelines, and microservice communication on GCP, designed to handle anything from a handful of messages to millions per second with no capacity planning.

Pub/Sub occupies an interesting middle ground: it's a pub/sub messaging system (like AWS SNS) and a streaming ingestion system (like Kafka/Kinesis) in one serverless service. It's the default way GCP services emit and consume events — Cloud Storage notifications, Cloud Functions triggers, dataflow pipelines — and a common front door for ingesting event streams into BigQuery and other analytics. Everything on the message queues concept page applies; this is the managed GCP implementation.

TL;DR

Pub/Sub is GCP's fully managed, globally-scalable messaging — publishers send to topics, subscriptions deliver to consumers, fully serverless (no capacity to manage, scales to millions of messages/sec).
It's both pub/sub messaging (fan-out to multiple subscribers) and streaming ingestion (high-throughput event pipelines) in one service.
Delivery: pull (consumers fetch, good for high throughput/control) or push (Pub/Sub POSTs to an HTTP endpoint / triggers Cloud Functions).
At-least-once delivery by default (design idempotent consumers); exactly-once and ordered delivery are available options.
Each subscription gets its own copy of a topic's messages — so multiple independent services can consume the same stream (fan-out built in).
vs Kafka: Pub/Sub is serverless with no partition management but less replay/streaming ecosystem; vs SNS/SQS: combines pub/sub and buffered delivery in one service.

Core Concepts

Topics and Subscriptions

A topic is a named channel publishers send messages to.
A subscription is attached to a topic and receives a copy of every message published to it. Multiple subscriptions on one topic mean multiple independent consumers each get the full stream — fan-out is built into the model.
Within a subscription, multiple consumer instances share the load (competing consumers), so you scale processing by adding consumers.

This design cleanly combines pub/sub (multiple subscriptions = multiple subscribers) with buffered queue-like delivery (a subscription holds messages until acknowledged).

Push vs Pull Delivery

Subscriptions deliver messages one of two ways:

Pull gives consumers control (fetch when ready, batch, throttle) — good for high-throughput workers. Push delivers proactively to an endpoint — natural for serverless (Cloud Functions triggered by a subscription) and simple HTTP consumers. Choose based on your consumer architecture.

Delivery Guarantees and Ordering

At-least-once (default) — every message is delivered at least once; on consumer failure or timeout (no ack), it's redelivered. So a message can arrive more than once — design idempotent consumers (the universal messaging rule).
Exactly-once delivery — an option that eliminates duplicates within a subscription (at some throughput cost) for workloads where duplicates are unacceptable.
Message ordering — off by default (messages can arrive out of order at scale); enable ordered delivery with an ordering key when sequence matters for related messages. Like Kafka partitions, ordering is per-key.
Dead-letter topics — messages that repeatedly fail (exceed a delivery-attempt limit) route to a dead-letter topic for inspection (poison-message handling).

Common Uses

Event-driven microservices — services publish events, others react (event-driven architecture).
Streaming ingestion — ingest high-volume events (clickstreams, IoT, logs) into BigQuery (via Dataflow or Pub/Sub BigQuery subscriptions) for analytics.
GCP service events — Cloud Storage notifications, Eventarc, and many GCP events flow through Pub/Sub.
Task/work distribution — buffer and distribute work to a pool of consumers.
Fan-out — one event delivered to many independent systems.

Pub/Sub vs Kafka vs SNS/SQS

Pub/Sub's advantage is serverless simplicity — no partitions, brokers, or capacity to manage, scaling automatically. Versus Kafka, it trades Kafka's rich replay and streaming ecosystem for zero ops (though Pub/Sub has retention/seek and a Kafka-compatible option). Versus AWS SNS/SQS, it combines pub/sub and buffered delivery in one service. On GCP, Pub/Sub is the default for both messaging and streaming ingestion; choose Kafka when you specifically need its streaming/replay depth.

Common Mistakes

Non-Idempotent Consumers

At-least-once delivery redelivers messages on failure/timeout, so the same message can be processed twice. Consumers that re-execute non-idempotently duplicate work. Design idempotent processing (dedupe by message ID), or enable exactly-once delivery if duplicates are truly unacceptable. The universal messaging rule.

Assuming Ordering by Default

Pub/Sub does not guarantee message order by default (messages can arrive out of order at scale). If sequence matters for related messages, enable ordered delivery with an ordering key. Assuming ordering without it is a common bug — the same consideration as Kafka partitions.

No Dead-Letter Topic

Without a dead-letter topic, a message that always fails is redelivered indefinitely (wasting resources) or eventually dropped. Configure a dead-letter topic with a max delivery-attempt limit so poison messages are captured for inspection.

Wrong Delivery Type

Using push for high-throughput workloads that need consumer-controlled batching (better with pull), or pull where a simple Cloud Functions trigger (push) would be simpler. Match delivery type to your consumer: pull for high-throughput workers, push for serverless/simple HTTP consumers.

Treating It as a Kafka Replacement for Replay-Heavy Workloads

Pub/Sub has retention and seek but not Kafka's full replay-by-offset streaming model. For workloads centered on replaying long event histories or complex stream processing, Kafka (or Pub/Sub's Kafka-compatible offering) may fit better. Use Pub/Sub for its serverless messaging/ingestion strengths; don't assume full Kafka parity.

FAQ

Is Pub/Sub messaging or streaming?

Both — it's a pub/sub messaging system (fan-out to multiple subscribers, buffered delivery) and a streaming ingestion system (high-throughput event pipelines feeding BigQuery/Dataflow). This dual nature is distinctive: on GCP it serves the roles that AWS SNS (pub/sub) and Kinesis/Kafka (streaming) play, in one serverless service.

Push or pull subscriptions?

Pull for high-throughput, consumer-controlled processing (workers fetch, batch, and ack at their own pace) — more control and efficiency at scale. Push for serverless and simple consumers — Pub/Sub POSTs messages to an HTTP endpoint or triggers a Cloud Function, so you don't run a polling consumer. Choose by consumer architecture; push is convenient for event-driven serverless, pull for dedicated high-volume workers.

How does Pub/Sub compare to Kafka?

Pub/Sub is fully serverless — no partitions, brokers, or capacity to manage, auto-scaling to millions of messages/sec. Kafka offers stronger replay (retained log, replay by offset) and a richer streaming ecosystem but requires managing (or paying for managed) partitions and brokers. For GCP event-driven messaging and ingestion with zero ops, Pub/Sub; for replay-heavy streaming and Kafka's ecosystem, Kafka. Pub/Sub also has a Kafka-compatible offering.

Does each consumer get every message?

Each subscription gets its own copy of every message on the topic — so multiple subscriptions mean multiple independent consumers each receive the full stream (fan-out). Within a single subscription, multiple consumer instances share the load (each message goes to one of them). So: fan-out across subscriptions, load-sharing within a subscription.

How do I get exactly-once processing?

Enable exactly-once delivery on the subscription (eliminates duplicates within it, at some throughput cost), and/or design idempotent consumers (dedupe by message ID). Default is at-least-once (duplicates possible on redelivery), so for workloads where duplicates are unacceptable, use exactly-once delivery plus idempotency for defense in depth.