gcp►b-data-engineer

b06-Cloud-PubSub

Cloud Pub/Sub

  • Global scale messaging buffer/coupler
  • No-ops
  • Decouples senders and receivers
  • Equivalent to Kafka
  • At-least-once delivery

Pub/Sub Core Concepts

  • Topic: A named resource to which messages are sent by publishers
  • Subscription: A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application.
  • Message: The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.
  • Message attribute: A key-value pair that a publisher can define

Pub/Sub Message flow

  1. Publisher sends messages to the topic
  2. Messages are stored in message store until they are delivered and acknowledged by subscribers
  3. Pub/Sub forwards messages from a topic to subscribers. messages can be pushed by Pub/Sub to subscriber or pulled by subscribers from Pub/Sub
  4. Subscriber receives pending messages from subscription and acknowledge to Pub/Sub
  5. After message is acknowledged by the subscriber, it is removed from the subscription’s queue of messages.

Basic flow of messages Message flow

Messages A, B are sent to the topic.
There are 2 subscriptions, Subscriber1 and Subscriber2 are subscribed to Subscription1 and each one got one message.
Subscriber3 is subscribed to Subscription2, and it got both message A and B

Publishing

  • Message format

    • Message data
    • Ordering Key
    • Attributes

  • Using Schema

    • Avro
    • Protocol Buffer

Receiving

Subscriber

  • At Least Once delivery

  • Retention duration
    Messages stored in PubSub
    10 minutes -> 7 days

  • Ack Deadline
    Messages to be deleted if not be consumed
    10 seconds (by default)

  • Expiration period
    Subscription to be deleted if no messages arrive
    31 days be (by default)

Delivery mode

  • Push = lower latency, more real-time
    • Push subscribers must be Webhook endpoints that accept POST over HTTPS
  • Pull ideal for large volume of messages - batch delivery

Replaying

Seeking to a timestamp

  • retain_acked_messages set to True
  • messagesRetentionDuration
    7 days by default

Seeking to a snapshot

define e snapshot, messages to replay are those who have not been consumed when creating of the
snapshot and the new ones.