Easy ReviewDOP-C02
Configuration Management and IaC18

Event processing patterns

Choose event routing, independent message copies, buffered work, or retained streams for replay.

Este conteúdo de estudo ainda não foi traduzido, então aparece em inglês. Ajude a traduzir

Match the event path to the processing need: routing selects destinations, fan-out gives applications copies, queues hold waiting work, and streams retain records for independent readers.

Routing, queuing, and streaming

Event processing separates the component that reports a change from the component that handles it. EventBridge routes events, SNS distributes copies, SQS buffers work, and Kinesis Data Streams retains records for independent readers.

Need Service or feature Mechanism
Route by event content EventBridge default or Classic event bus Rules match patterns and select targets
Connect one source to one target EventBridge Pipes Filter, optionally enrich, then deliver
Distribute copies SNS Push a message to subscriptions
Buffer work SQS ConsumersA consumer is an application that reads and processes records from a stream. poll and delete completed messages
Read and replay a record sequence Kinesis Data Streams Consumers read retained records from shardsA shard is an ordered sequence of records within a Kinesis data stream.
Deliver records to storage or analytics Firehose Buffer records and deliver to a configured destination

A shard is one ordered sequence within a Kinesis stream. A consumer is an application that reads or processes messages or records.

How it works

  1. Choose routing when event content determines the destination. A default or Classic event bus compares the event with rule patterns.
  2. Choose SNS when subscribers need copies. Add an SQS queue to a subscription when that consumer needs stored work to poll later.
  3. Choose Kinesis Data Streams when applications need separate read positions and replay. Reading does not consume the retained record for other applications.
  4. For managed destination delivery, let Firehose consume Kinesis Data Streams. It can write batches to S3 or index records in OpenSearch. A Lambda consumer can independently process the same stream.

Exam cues

  • If the question says several applications must reread the original events independently, think Kinesis Data Streams, because each application can read retained records.
  • If the question says filter and enrich one source before sending to one target, think EventBridge Pipes, because that is its integration model.
  • If the question says distribute one notification but let each application work later, think SNS with SQS subscriptions, because SNS sends copies and SQS stores them.

Remember

  • Kinesis retention defaults to 24 hours and can reach 365 days.
  • SQS messages need deletion after successful processing.
  • Firehose buffers destination delivery. Redshift, S3, and OpenSearch are destination choices, not independent replay consumers.
  • Firehose OpenSearch input must be UTF-8 single-line JSON. Athena queries data stored in S3.

Trap: A delivery pipeline alone does not explain how another application will replay the source. Keep the source in Kinesis Data Streams when independent rereading is required.

Stream, deliver and query operational data
write1consume2deliver3or index4poll5Producersapps, CloudWatch LogsiData Streamsretain + replayiFirehosebuffer + deliveriS3archive + AthenaiOpenSearchsearch + dashboardsiLambdacustom consumeri
  1. Producers write records to the stream with partition keys.

  2. Firehose reads the stream and buffers records for delivery.

  3. An S3 destination stores batches as objects that Athena can query.

  4. An alternative OpenSearch destination indexes the records in bulk.

  5. A Lambda or KCL consumer can read the same stream independently for custom processing.

Detalhes das caixas (6)
Producers

A producer writes records with partition keys to Kinesis Data Streams.

Data Streams

Records remain available during retention. Multiple applications can consume the same stream independently.

Firehose

Firehose consumes the stream and buffers records before destination delivery.

S3

Firehose stores batches as S3 objects. Athena can query data stored in S3.

OpenSearch

Firehose uses bulk requests to index records in OpenSearch. Input records must be UTF-8 single-line JSON objects.

Lambda

A Lambda event source mapping reads Kinesis records and invokes a function. Duplicate processing is possible.

Da documentação da AWS (15)
  • An EventBridge default or Classic event bus evaluates incoming events against rule patterns and sends matches to the configured targets.
  • An EventBridge pipe connects one supported source to one target, with optional filtering followed by enrichment before delivery.
  • SNS pushes published messages to subscriptions; SQS stores messages for applications to retrieve by polling.
  • Receiving an SQS message hides it temporarily; the consumer deletes it after successful processing.
  • Kinesis Data Streams stores sequences of records in shards, and multiple applications can read a stream independently.
  • Kinesis Data Streams retains records for 24 hours by default and supports retention up to 365 days.
  • Firehose delivers records to a configured destination, including S3, Redshift, OpenSearch, Splunk, HTTP endpoints, or Snowflake.
  • Firehose buffers incoming records before delivery; an S3 destination combines records into objects.
  • A Firehose delivery stream can act as a consumer of Kinesis Data Streams.
  • Firehose sends bulk indexing requests to OpenSearch and expects UTF-8 single-line JSON records.
  • Lambda event source mappings read Kinesis records and can process records more than once.
  • Athena can query data stored in Amazon S3 using SQL.
  • A Kinesis partition key groups records into a shard; each shard holds a sequence of records.
  • CloudWatch Logs subscription filters can forward log events to Kinesis Data Streams.
  • Kinesis Client Library supports fault-tolerant consumption of stream records.

Durable fan-out

Fan-outFan-out distributes copies of a published message to multiple subscribers. means sending a copy of one message to multiple destinations. SNS with one SQS queue per application gives each application stored work that it can process at its own pace.

How it works

  1. Subscribe a separate SQS queue for each application to the SNS topic. A queue is the application’s backlogA backlog is queued work that consumers have not finished processing., meaning work waiting to be processed.
  2. Allow SNS to send messages in each queue’s resource policyA resource policy is a permission document attached to a resource to control who can access it., the permission document attached to that queue. Restrict the permission to the intended topic with aws:SourceArn, its Amazon Resource NameAn Amazon Resource Name identifies a specific AWS resource., or unique AWS resource identifier.
  3. Publish to SNS. By default each subscription receives a copy. A subscription filter can select only matching attributes or JSON body fields, the named values in a structured text message.
  4. Each application polls its own queue. Configure retention and a dead-letter queueA dead-letter queue holds messages that could not be processed successfully for investigation., a separate queue for repeatedly unsuccessful messages, according to that application’s needs.
  5. For DynamoDB changes, one Lambda stream consumerA consumer is an application that reads and processes records from a stream. can publish to SNS. This reduces direct readers when more than two readers per shardA shard is an ordered sequence of records within a Kinesis data stream. cause throttling. EventBridge rules can also route matching events to SNS, as the diagram shows.

Exam cues

  • If the question says every application needs every message despite temporary downtime, think SNS with an SQS queue per application, because each gets stored work.
  • If the question says additional DynamoDB Streams readers cause throttling, think one forwarding Lambda consumer and downstream fan-out, because adding readers can exceed the per-shard reading recommendation.
  • If the question says an SNS subscription exists but SQS delivery is denied, think queue resource policy, because subscription setup alone does not grant send permission.

Remember

  • SQS retention defaults to 4 days; its configurable range is 1 minute to 14 days.
  • DynamoDB Streams should have no more than two simultaneous readers per shard.
  • SNS subscriptions receive every message by default; filtering is configured per subscription.
  • A queue’s maxReceiveCount controls movement to its dead-letter queue.

Trap: A queue shared by applications does not give each application its own backlog. Use a separate SNS subscription and queue for each application that needs its own copy.

Route events and buffer independent consumers
emit1match2publish3publish4Event sourceiEventBridgerule matches eventiSNS topicfan-outiSQS queue Aconsumer AiSQS queue Bconsumer Bi
  1. A producer sends an event to the bus.

  2. An EventBridge rule matches the event and sends it to an SNS topic.

  3. SNS delivers a copy to queue A for its consumer.

  4. SNS delivers another copy to queue B, allowing independent processing.

Detalhes das caixas (5)
Event source

An application or AWS service emits an event to the event bus.

EventBridge

A default or Classic event bus rule sends matching events to an SNS target.

SNS topic

SNS pushes a copy to each matching subscription.

SQS queue A

This queue stores messages for its own polling consumer, subject to its configured retention.

SQS queue B

This queue stores messages for its own polling consumer, subject to its configured retention.

Da documentação da AWS (10)
  • Subscribing separate SQS queues to an SNS topic gives each subscribed queue a copy of published messages for later processing.
  • An SQS queue policy must allow sns.amazonaws.com to call sqs:SendMessage; aws:SourceArn can restrict that permission to a specific topic.
  • An SNS subscription receives every published message by default; a filter policy selects messages using attributes or the JSON body.
  • SQS message retention defaults to 4 days and can be configured from 1 minute to 14 days.
  • AWS recommends no more than two simultaneous readers per DynamoDB Streams shard because additional readers can cause throttling.
  • One Lambda consumer can forward DynamoDB Streams records to SNS, which delivers copies to subscribed SQS queues.
  • An EventBridge default or Classic event bus rule can route a matching event to an SNS topic.
  • SQS can move repeatedly unsuccessful messages to a dead-letter queue according to the source queue redrive policy maxReceiveCount.
  • An Amazon Resource Name uniquely identifies an AWS resource.
  • A queue resource policy controls whether SNS is allowed to send messages to that SQS queue.

Backpressure and idempotency

BackpressureBackpressure occurs when producers consistently send messages faster than consumers can process them. occurs when a producer consistently sends work faster than its consumerA consumer is an application that reads and processes records from a stream. can process it. An SQS backlogA backlog is queued work that consumers have not finished processing. holds waiting work while a Lambda event source concurrency limit controls simultaneous processing. IdempotencyIdempotency lets a consumer process the same message repeatedly without unintended extra effects. means repeated processing has no unintended extra effect.

How it works

  1. Buffer work in SQS. Monitor visible messages and the approximate age of the oldest unprocessed message to detect a growing backlog. To increase processing capacity, investigate function speed, batch size, concurrency quotas, or provisioned polling.
  2. For an SQS Lambda mapping without provisioned mode, set maximum concurrency to limit simultaneous invocations. Reserved concurrencyReserved concurrency sets the simultaneous executions reserved for and allowed on a Lambda function. is the function-level allocation; keep it sufficient for the sum of its SQS mapping limits.
  3. Receiving a message starts its visibility timeoutThe visibility timeout temporarily hides a received SQS message while a consumer processes it., the period during which it is hidden from other receives. Delete it after success. Undeleted messages can return after that period.
  4. Make the handler idempotent because processing can repeat, including with FIFO, meaning first in, first out, queues.
  5. Use a redrive policy to move repeatedly unsuccessful messages to a dead-letter queueA dead-letter queue holds messages that could not be processed successfully for investigation. for investigation.

Exam cues

  • If the question says a queue-triggered function overwhelms its downstream service, think maximum concurrency on the mapping, because it caps simultaneous invocations from that queue.
  • If the question says repeated messages must not cause extra effects, think idempotent processing, because standard queues and Lambda SQS mappings can process messages more than once.
  • If the question says a payment succeeded before the handler crashed, think idempotent handling, because deleting or acknowledging work can fail after its side effect succeeds.

Remember

  • FIFO order is per message groupA message group identifies the messages that an SQS FIFO queue processes in order.; different groups can be processed independently.
  • Reserved concurrency should cover the sum of maximum concurrency values for the function’s SQS mappings.
  • maxReceiveCount controls SQS movement to a dead-letter queue.

Trap: FIFO deduplicationDeduplication prevents repeated producer sends from adding duplicate messages to a FIFO queue within its deduplication interval. removes repeated producer sends within a 5-minute interval. It does not make a Lambda handler’s side effects execute exactly once: Lambda SQS mappings can process a message more than once.

Da documentação da AWS (12)
  • SQS hides a received message for the visibility timeout; if it is not deleted, it can be received again after that timeout.
  • SQS standard queues can deliver a message more than once; consumers should tolerate repeated processing without unintended effects.
  • Lambda SQS event source mappings process messages at least once, including FIFO sources, so function code must handle duplicate processing.
  • For an SQS event source mapping without provisioned mode, maximum concurrency limits simultaneous Lambda invocations from that queue.
  • If a Lambda function has reserved concurrency, it should be at least the sum of maximum concurrency values for all its SQS mappings to avoid throttling.
  • SQS FIFO ordering applies within a message group; different message groups can be processed independently.
  • The SQS redrive policy maxReceiveCount controls how many receives a message can have before movement to a dead-letter queue.
  • SQS ApproximateNumberOfMessagesVisible estimates available work and ApproximateAgeOfOldestMessage reports the oldest unprocessed message age, with documented exclusions for repeatedly received standard-queue messages.
  • Backpressure occurs when an SQS producer persistently sends messages faster than the Lambda consumer processes them.
  • To increase Lambda SQS processing capacity, investigate function performance, batch size, concurrency quota, or provisioned event-source polling.
  • SQS FIFO avoids adding duplicate messages when the same SendMessage is retried within its 5-minute deduplication interval.
  • Reserved concurrency reserves a function-level execution allocation and sets its upper concurrency limit.

Lambda SQS batch failures

Lambda can process SQS messages in batches. Partial batch responses let failed messages be retried without retrying the successful messages in the same batch.

How it works

  1. Set the SQS visibility timeoutThe visibility timeout temporarily hides a received SQS message while a consumer processes it. to at least six times the Lambda function timeout, plus the batching window when configured. See queue configuration.
  2. Enable ReportBatchItemFailures on the event source mapping and return failed message IDs in batchItemFailures. By default, a batch failure retries every message in the batch. See partial batch responses.
  3. For a FIFO queue, stop processing after the first failure and report both failed and unprocessed messages to preserve order.

Exam cues

  • If the question says one message fails in a batch of successful messages, think partial batch responses, because they retry only the reported failures.
  • If the question says an SQS batching window is configured, think visibility timeout of at least six times the function timeout plus the window, because the recommendation accounts for both.
  • If the question says a FIFO batch has a failure, think stop at the first failure and report failed and unprocessed messages, because this preserves order.

Remember

  • Visibility timeout: at least 6 times the function timeout, plus any configured batching window.
  • Enable ReportBatchItemFailures and return failed message IDs in batchItemFailures.
  • For FIFO, include unprocessed messages in the response after the first failure.

Trap: Enabling partial batch responses alone does not identify failed messages. The function must return their IDs in batchItemFailures; for FIFO, it must also report unprocessed messages after the first failure.

Da documentação da AWS (3)
  • A Lambda SQS batch failure retries the entire batch by default; ReportBatchItemFailures with a batchItemFailures response retries only reported failed messages.
  • AWS recommends SQS visibility timeout of at least six times the Lambda function timeout, plus the batch window when one is configured.
  • For a FIFO partial batch response, stop after the first failed message and report failed and unprocessed messages to preserve ordering.

Flashcards

toque para virar
Resposta

·

Todos os 18 flashcards em lista

When should an application use Kinesis Data Streams rather than Firehose?

Use Kinesis Data Streams when independent consumers must reread retained records. Use Firehose for buffered delivery to a configured destination.

A team archives transformed events through Firehose and also needs independent replay of the original events. Which source should Firehose consume?

Kinesis Data Streams. It retains the source records so another consumer can reread them while Firehose handles destination delivery.

How do EventBridge event buses, SNS, and SQS differ in an event processing design?

EventBridge default or Classic bus rules route matching events. SNS pushes messages to subscribers. SQS buffers messages for polling consumers.

How does processing an SQS message differ from reading a Kinesis Data Streams record?

An SQS consumer deletes a successfully processed message. A Kinesis record remains available during retention for independent readers.

An application must filter one SQS source, add reference data, and send matching events to one target without integration code. Which EventBridge feature fits?

EventBridge Pipes. A pipe connects one source to one target with optional filtering and enrichment.

An operations team must route only matching AWS service events to an automation target. Which EventBridge mechanism fits?

A rule on the default event bus. EventBridge evaluates the rule pattern and delivers matching events to its targets.

Several applications reading a DynamoDB stream are throttled. How can they receive changes without each reading the stream?

Use one Lambda stream consumer to publish to SNS and subscribe one SQS queue per application. More than two readers per DynamoDB Streams shard can cause throttling.

An SNS publisher must send each event to three applications that can be offline temporarily. What pattern gives each application stored work?

Subscribe one SQS queue per application to the SNS topic. Each queue holds its own copy within its retention period.

A team adds application readers to DynamoDB Streams and encounters read throttling. How can it distribute changes with fewer readers?

Have one Lambda consumer forward changes to SNS and subscribe separate SQS queues. This avoids one direct stream reader per downstream application.

Two applications need every SNS message but process at different rates. How should their SQS subscriptions be arranged?

Use one SQS queue per application. Each queue stores a separate copy; configure retention and failure handling for each queue.

An SNS topic subscription exists, but its SQS queue rejects delivery. Which permission should the team check?

The SQS queue policy must allow sns.amazonaws.com to perform sqs:SendMessage. Restrict aws:SourceArn to the intended topic.

An SNS subscriber needs only selected event types. What should the team configure?

A filter policy on that subscription, matching message attributes or the JSON body. Without a filter, the subscription receives every published message.

An SQS-triggered Lambda function overwhelms a downstream database during traffic bursts. Which event source setting can limit simultaneous work?

Set maximum concurrency on the SQS event source mapping without provisioned mode. Keep function reserved concurrency sufficient for all configured SQS mapping limits.

An SQS-triggered Lambda function finishes a payment operation but fails before the message is deleted. Can FIFO alone prevent the operation from repeating?

No. Lambda SQS mappings can process a message more than once, including FIFO sources. Make the payment handler idempotent so a repeated message has no extra effect.

What does idempotent processing mean for an SQS consumer?

Processing the same message again causes no unintended extra effect. Standard queues and Lambda event source mappings can deliver duplicate work.

A Lambda SQS batch contains successful messages and one failure. How can the team retry only failed messages?

Enable ReportBatchItemFailures and return failed message IDs in batchItemFailures. By default a batch failure returns every message for retry.

A Lambda function uses an SQS batching window. What visibility timeout does AWS recommend?

At least six times the function timeout, plus the configured batching window. This leaves time for processing and throttling retries.

A Lambda function processes an SQS FIFO batch and encounters a failed message. How should its partial batch response preserve order?

Stop processing after the first failure and report both failed and unprocessed messages in batchItemFailures.

Você concluiu todos os tópicos da DOP-C02!

Seu badge ficou dourado. Próximo passo: agendar a prova na AWS. Depois de passar, adicione-se ao Quem passou.

Compartilhar

Conte que você concluiu esta certificação

Concluí todos os tópicos da AWS Certified DevOps Engineer - Professional (DOP-C02) no Easy Review. Venha estudar também!