Easy ReviewDOP-C02
Monitoring and Logging25

Kinesis Data Streams

Retain streaming records for independent consumers, with explicit capacity, ordering, and replay choices.

Este conteúdo de estudo ainda não foi traduzido, então aparece em inglês. Ajude a traduzir

Kinesis Data Streams divides retained records into shards. Producers choose partition keys, consumers choose read positions, and checkpoints let applications resume processing after interruption.

Shards, partition keys, and capacity

Kinesis Data Streams stores records for applications to process independently. A shardA shard is an ordered record sequence that provides a unit of stream capacity. is an ordered lane of records with its own capacity. The partition keyA partition key groups records by mapping them to a shard through a hash value. is the producer’s grouping value that normally decides which lane receives a record.

How it works

  1. A producer sends record bytes with a partition key. Kinesis hashes that key using MD5An MD5 hash maps a partition key to a number used to choose a shard hash range., a function that maps it to a number, and chooses the shard whose hash range contains it.
  2. Distribute traffic across keys. A hot keyA hot key concentrates traffic on one shard until the shard cannot accept more work., meaning one key carrying too much traffic, can throttle its shard even while other shards have room.
  3. Choose capacity management. Provisioned mode requires a shard count; on-demand manages shards automatically. On-demand can still throttle on a hot key or a sudden increase beyond its scaling envelope.
  4. Choose a maximum record size that downstream consumers can handle. Protect stored data with KMS server-side encryptionServer-side encryption protects record data stored in Kinesis using AWS KMS keys. and API connections with Transport Layer SecurityTLS encrypts network connections used to access Kinesis APIs., or TLS, which encrypts data in transit.
Mode Capacity responsibility Cue
Provisioned Choose shard count; split or merge shards Manage known capacity explicitly
On-demand Standard Kinesis manages shards Variable, unpredictable traffic
On-demand Advantage Account-level setting for on-demand streams Proactively warm write capacity

Exam cues

  • If the question says most shards are idle but writes throttle, think hot partition key, because the same key maps to one shard.
  • If the question says unpredictable traffic with no shard management, think On-demand Standard, because it automatically manages capacity.
  • If the question says server log lines must reach a stream without changing the application, think Kinesis Agent, because it watches log files and sends new lines to the stream.

Remember

  • Sustained shard writes: 1 MB/s or 1,000 service records/s. Shared reads: 2 MB/s and 5 GetRecords calls/s.
  • New on-demand stream: 4 MB/s writes and 8 MB/s reads.
  • On-demand supports twice the prior 30-day peak; exceeding that within 15 minutes can throttle.
  • October 2025: AWS introduced records up to 10 MiBA mebibyte is a unit equal to 1,048,576 bytes.. The default maximum remains 1 MiB; UpdateMaxRecordSize accepts 1024 to 10240 KiB.
  • Large records use intermittent burst capacityBurst capacity temporarily accepts intermittent large records above the sustained stream rate.. Check Region support and downstream limits, including Firehose destination restrictions.
  • API connections require TLS 1.2; AWS recommends TLS 1.3.

Trap: A larger allowed record does not create a higher sustained shard write rate. Separate the per-record size setting from sustained capacity.

Partition keys distribute records among shards
hash range A1hash range B2ProduceriShard AiShard Bi
  1. A partition key hash selects shard A when it falls in that range.

  2. A hash in another range selects shard B; one hot key stays on its mapped shard.

Detalhes das caixas (3)
Producer

Each record carries a partition key. Its MD5 hash maps it to a shard.

Shard A

Records whose key hashes into this range use this shard and its capacity.

Shard B

A different hash range can use a different shard. There is no order across shards.

Da documentação da AWS (20)
  • A Kinesis data stream contains shards; a record contains a partition key, assigned sequence number, and data bytes.
  • Kinesis uses an MD5 hash of the partition key to map records to a shard hash range; the same key maps to the same shard unless explicitly overridden.
  • A shard supports sustained writes up to 1 MB per second; intermittent large records use burst capacity.
  • Shared GetRecords consumers collectively have 2 MB per second per shard and up to 5 read transactions per second.
  • On-demand streams can still throttle when one partition key exceeds the shard limit; spreading traffic across keys avoids concentrating it on one shard.
  • Provisioned mode requires a chosen shard count and charges by shard-hour.
  • On-demand Standard automatically manages shard capacity and suits variable, unpredictable traffic.
  • On-demand Advantage is an account-level setting for on-demand streams and supports proactively warming write capacity.
  • New on-demand streams start with 4 MB per second write and 8 MB per second read throughput.
  • On-demand accommodates up to twice the prior 30-day peak write throughput; increases beyond twice that peak within 15 minutes can throttle.
  • The default maximum record size is 1 MiB; the stream setting can be increased up to 10 MiB in supported Regions.
  • UpdateMaxRecordSize accepts MaxRecordSizeInKiB values from 1024 to 10240 KiB, equivalent to 1 to 10 MiB.
  • AWS announced support for record sizes up to 10 MiB in October 2025.
  • Kinesis Data Streams supports server-side encryption of stored data with AWS KMS keys.
  • Kinesis API clients must support TLS 1.2; AWS recommends TLS 1.3 for connections.
  • Increasing the Kinesis maximum record size requires checking downstream limits; Firehose supports large-record S3 delivery but not dynamic partitioning, HTTP, Snowflake, or Redshift delivery.
  • A shard supports writes of up to 1000 Kinesis service records per second.
  • Resharding splits or merges shards to change provisioned stream capacity.
  • Amazon Kinesis Data Streams collects and processes large streams of data records in real time.
  • Kinesis Agent is a Java application for Linux servers that watches log files and sends new lines to a Kinesis data stream.

Retention and replay

RetentionThe retention period is how long a written record remains accessible in the stream. is how long Kinesis keeps a record available after it is written. ReplayReplay reads earlier retained records again for processing. means reading retained records again from an earlier position. Separate consumer applications can read the same stream independently without changing the stored records.

How it works

  1. Choose retention long enough for a consumer to recover from bugs or a downstream outage. The default and minimum are 24 hours; the maximum is 365 days.
  2. Keep in mind that a stored record is immutable: its bytes and order cannot be changed after writing.
  3. Select a starting position for each shardA shard is an ordered record sequence that provides a unit of stream capacity.. A shard iteratorA shard iterator is a short-lived token identifying where a shard read begins. is the token that identifies that read position for the low-level API.
  4. Call GetRecords and continue with its returned NextShardIterator. The iterator expires after 5 minutes, so it is a read token rather than long-term saved progress.
  5. Replay while the required data is retained. Reducing retention can make older records inaccessible almost immediately.
Iterator type Starting point
TRIM_HORIZON Oldest retained record
LATEST Just after the most recent record
AT_TIMESTAMP Specified time, bounded by retained data
AT_SEQUENCE_NUMBER Includes the specified record
AFTER_SEQUENCE_NUMBER Immediately after the specified record

Exam cues

  • If the question says replay everything still available after a bug fix, think TRIM_HORIZON, because it starts at the oldest retained record.
  • If the question says process only events arriving after the reader starts, think LATEST, because it skips the retained history.
  • If the question says recover from an outage longer than the default window, think longer retention before records expire, because consumers can recover only while data remains accessible.
  • If the question says a DynamoDB Streams Kinesis Adapter extends retention, think the original DynamoDB Streams source, because the adapter only redirects the consumer interface.

Remember

  • Kinesis retention: 24 hours by default and minimum; 8760 hours or 365 days maximum.
  • Retention beyond 24 hours costs extra.
  • Shard iterators expire after 5 minutes.
  • AT_TIMESTAMP older than the retention boundary starts at the oldest retained record.
  • DynamoDB Streams still retains change records for 24 hours when read through its Kinesis Adapter.

Trap: Reducing retention can remove access to data immediately. A later replay position cannot recover records that are already outside the retention window.

Da documentação da AWS (13)
  • Kinesis Data Streams retention defaults to 24 hours, has a minimum of 24 hours, and can be increased up to 8760 hours or 365 days.
  • Kinesis retention beyond 24 hours incurs additional charges.
  • After a record is written to Kinesis Data Streams, its contents and order within the stream cannot be modified.
  • Multiple applications can consume the same Kinesis stream independently and concurrently.
  • TRIM_HORIZON starts at the oldest record still retained in the shard.
  • LATEST starts just after the most recent shard record and reads newly arriving records.
  • AT_TIMESTAMP starts from a specified timestamp; a timestamp older than retained data starts at the oldest retained record.
  • AT_SEQUENCE_NUMBER includes the specified sequence number; AFTER_SEQUENCE_NUMBER starts immediately after it.
  • A shard iterator is a read-position token that expires 5 minutes after creation; GetRecords returns NextShardIterator for the following read.
  • Reducing retention makes records older than the new retention period inaccessible almost immediately.
  • Retention must allow time for consumers to recover from processing bugs or downstream outages before records expire.
  • The DynamoDB Streams Kinesis Adapter directs KCL API calls to DynamoDB Streams; it does not change that source into Kinesis Data Streams.
  • DynamoDB Streams retains change records for 24 hours.

Enhanced fan-out and checkpoints

Enhanced fan-outEnhanced fan-out pushes records to each registered consumer with dedicated read throughput per shard. gives each registered consumer its own shardA shard is an ordered record sequence that provides a unit of stream capacity. read throughput. Kinesis Client Library, or KCL, coordinates workers and stores read progress. These solve separate problems: read isolation controls capacity, while checkpointingA checkpoint stores the last completed record position so a consumer can resume processing. records completed work.

How it works

  1. Choose shared reads when applications can share the shard’s 2 MB/s through GetRecords polling, meaning repeated requests for new records. Choose enhanced fan-out when each application needs dedicated read throughput.
  2. Register the enhanced fan-out consumer with RegisterStreamConsumer. SubscribeToShard pushes records over HTTP/2 with up to 2 MB/s per consumer per shard.
  3. If using KCL, let its workers coordinate shard ownership through a leaseA lease records which KCL worker currently owns processing of a shard., the stored assignment of a shard to a worker. The DynamoDB lease table also stores each shard’s checkpoint, its saved completed position.
  4. Complete the processing side effect before checkpointing that progress. If the worker crashes between those operations, its replacement starts from the last checkpoint and can repeat the side effect.
  5. Make processing idempotent so a repeated record has no unintended extra effect.

Exam cues

  • If the question says readers compete for the same shard throughput, think enhanced fan-out, because each registered consumer has independent capacity.
  • If the question says processing completed before a worker crashed but checkpointing did not, think repeated processing after restart, because the saved position still points behind the completed work.

Remember

  • Shared reads: 2 MB/s total per shard. Enhanced fan-out: up to 2 MB/s per consumer per shard.
  • Standard and Provisioned: up to 20 registered enhanced fan-out consumers per stream.
  • November 2025: AWS announced up to 50 for On-demand Advantage in supported Regions.

Trap: Enhanced fan-out does not make processing exactly once. Dedicated read capacity and saved checkpoints still require duplicate-safe side effects.

Independent consumers keep separate read progress
push1push2checkpoint A3checkpoint B4Kinesis streamiKCL app AiKCL app BiLease metadatai
  1. Consumer A receives its own enhanced fan-out stream of records.

  2. Consumer B independently receives the same records.

  3. Application A saves completed progress in its own lease metadata.

  4. Application B saves its own progress independently.

Detalhes das caixas (4)
Kinesis stream

Each enhanced fan-out consumer has dedicated throughput up to 2 MB/s per shard.

KCL app A

This application independently processes the stream and checkpoints its progress.

KCL app B

Another application can process the same records independently.

Lease metadata

Each KCL application maintains its own DynamoDB lease table with shard ownership and checkpoints.

Da documentação da AWS (10)
  • Shared-throughput consumers poll GetRecords and share 2 MB per second of read throughput per shard.
  • An enhanced fan-out consumer receives dedicated throughput up to 2 MB per second per shard, independent of other consumers.
  • Enhanced fan-out pushes records over HTTP/2 with SubscribeToShard instead of GetRecords polling.
  • Enhanced fan-out consumers are registered for a stream using RegisterStreamConsumer and subscribe to shards with SubscribeToShard.
  • On-demand Standard and Provisioned streams support up to 20 registered enhanced fan-out consumers per stream; On-demand Advantage supports up to 50 in supported Regions.
  • AWS announced support for 50 enhanced fan-out consumers on On-demand Advantage streams in November 2025.
  • KCL uses a DynamoDB lease table to track which worker processes a shard and the most recent checkpoint sequence number.
  • When a KCL processor restarts after worker failure, deployment, or resharding, it resumes from the last checkpoint and can repeat completed processing.
  • A checkpoint records completed progress; a crash after a side effect but before checkpointing can cause that side effect to be attempted again.
  • Consumers should handle records more than once without unintended extra side effects; this is idempotent processing.

Producer batching and ordering

Kinesis Data Streams producers choose how to route and write records. The write method and Kinesis Producer Library (KPL) batching settings affect ordering and producer throughput.

How it works

  1. Kinesis uses an MD5 hashAn MD5 hash maps a partition key to a number used to choose a shard hash range. of the partition keyA partition key groups records by mapping them to a shard through a hash value., a value supplied with a record, to choose a shardA shard is an ordered record sequence that provides a unit of stream capacity. by its hash range. The same key maps to the same shard unless the mapping is explicitly overridden.
  2. For strict write order, send PutRecord calls serially to the same shard and use SequenceNumberForOrdering. The ordering guarantee applies to writes from the same client and partition key; sequence numbersA sequence number identifies a record position within its shard. do not establish order across shards.
  3. KPL aggregationAggregation combines multiple application records inside one Kinesis record. packs multiple application records into one Kinesis record, reducing the service-record rate and improving producer throughput.
  4. KPL collectionCollection sends multiple Kinesis records in one PutRecords request. sends multiple Kinesis records in one PutRecords request. Aggregation and collection can be configured independently.
  5. A PutRecords request can partially succeed. Check for failed entries and retry them; the operation does not guarantee record ordering.

Exam cues

  • If the question says many small application records reach the shard record-rate limit, think KPL aggregation, because it packs them into fewer Kinesis records.
  • If the question says strict write order matters, think serial PutRecord calls with SequenceNumberForOrdering, because PutRecords does not guarantee ordering.
  • If the question says a PutRecords batch has failures, think retry failed entries, because other entries may have succeeded.

Remember

  • Aggregation combines application records; collection sends Kinesis records together using PutRecords.
  • Both KPL batching features are enabled by default and can be configured independently.
  • Sequence numbers do not establish order across shards.
  • KCL 3.x keeps leaseA lease records which KCL worker currently owns processing of a shard., worker metrics and coordinator state tables; KCL 3.5 supports one lease table.
  • Enhanced fan-outEnhanced fan-out pushes records to each registered consumer with dedicated read throughput per shard. propagation typically averages 70 ms; it is not a per-record guarantee, and it does not make processing exactly once.
  • Producer retries can duplicate payloads under new sequence numbers; deduplicate on an application identifier.

Trap: A PutRecords request can succeed for some entries and fail for others. A batch does not guarantee strict ordering, even when records are sent together.

Da documentação da AWS (13)
  • Kinesis uses an MD5 hash of the partition key to map records to a shard hash range; the same key maps to the same shard unless explicitly overridden.
  • For strict write ordering, write PutRecord calls serially to one shard and use SequenceNumberForOrdering; sequence numbers do not establish order across shards.
  • KPL aggregation combines multiple application records into one Kinesis record to reduce the service-record rate and improve producer throughput.
  • KPL collection uses PutRecords to send multiple Kinesis records in one request; aggregation and collection are independently configurable and enabled by default.
  • PutRecords can partially succeed and does not guarantee record ordering; retry failed entries rather than assuming the whole request succeeded.
  • KCL 3.x uses lease, worker metrics, and coordinator state DynamoDB tables by default; KCL 3.5 supports consolidating metadata into one lease table.
  • New migrations from KCL 2.x to KCL 3.5 use single table format by default; existing KCL 3.x applications require a migration to consolidate tables.
  • A producer retry after an uncertain acknowledgement can store duplicate payloads with different sequence numbers; an application-level key can identify duplicates.
  • An enhanced fan-out consumer receives dedicated throughput up to 2 MB per second per shard, independent of other consumers.
  • When a KCL processor restarts after worker failure, deployment, or resharding, it resumes from the last checkpoint and can repeat completed processing.
  • Consumers should handle records more than once without unintended extra side effects; this is idempotent processing.
  • AWS describes enhanced fan-out message propagation delay as typically averaging 70 milliseconds.
  • The KCL publishes CloudWatch metrics under the application name, including MillisBehindLatest per shard, which shows how far consumers lag.

Flashcards

toque para virar
Resposta

·

Todos os 25 flashcards em lista

What sustained write and shared-read capacity does one Kinesis Data Streams shard provide?

Writes: 1 MB/s or 1000 records/s. Shared reads: 2 MB/s with up to 5 GetRecords calls/s. Large-record burst support does not raise the sustained rate.

A Kinesis Data Streams stream throttles writes while most shards are idle. What should the team investigate?

A hot partition key can overload one shard. Distribute records across more keys; on-demand capacity or extra shards does not remove a single key hotspot.

How do Kinesis Data Streams Provisioned and On-demand Standard capacity modes differ?

Provisioned requires choosing and managing shard capacity and charges by shard-hour. On-demand Standard manages shards automatically and suits unpredictable traffic.

A Kinesis application needs occasional 4 MiB records. Is the default stream setting enough, and does enabling large records raise sustained shard throughput?

Raise the maximum record-size setting from its 1 MiB default, up to 10 MiB in supported Regions. Large records use burst capacity; sustained shard limits remain unchanged. Check downstream compatibility.

An on-demand Kinesis stream receives more than twice its recent peak within 15 minutes and throttles. Can on-demand scaling eliminate this case?

No. On-demand accommodates up to twice the previous 30-day peak; faster increases can throttle and require retries.

A company needs to collect new log lines from Linux servers for real-time stream processing. Which Amazon Kinesis option fits?

Use Kinesis Agent to send new log lines to Amazon Kinesis Data Streams for real-time processing.

A company must ingest and process website clickstream data in real time. Which AWS service fits?

Amazon Kinesis Data Streams.

Web servers write log files, and the lines must reach a Kinesis data stream without changing the application. What tool does that?

Kinesis Agent, installed on the servers and configured with the files to watch and the target stream.

What are the default, minimum, and maximum Kinesis Data Streams retention periods?

Default and minimum: 24 hours. Maximum: 8760 hours, or 365 days. Retention beyond 24 hours incurs additional charges.

A team fixes a Kinesis consumer bug and must replay all records still retained. Which iterator type should it request?

TRIM_HORIZON. It starts at the oldest record still retained in each shard. LATEST reads new arrivals instead of the retained history.

A Kinesis consumer should ignore retained history and process only newly arriving records. Which iterator type fits?

LATEST. It starts just after the most recent record. Use AT_TIMESTAMP or TRIM_HORIZON when retained history must be read.

A team reduces Kinesis retention while a consumer is far behind. What happens to records older than the new window?

They become inaccessible almost immediately. Size retention for recovery before shortening it.

Can a Kinesis Data Streams consumer edit a stored record to correct a payload before replaying it?

No. Written record contents and order are immutable. Replay rereads retained records; the consumer must handle the original data.

Does the DynamoDB Streams Kinesis Adapter provide Kinesis Data Streams retention for DynamoDB change records?

No. It directs KCL calls to DynamoDB Streams, whose retention is 24 hours. The adapter changes the interface used by the consumer, not the underlying source.

A low-level Kinesis consumer needs to resume immediately after a known sequence number. Which iterator type should it use?

AFTER_SEQUENCE_NUMBER. AT_SEQUENCE_NUMBER includes that record. Continue subsequent reads with NextShardIterator from GetRecords.

Five Kinesis Data Streams applications each need the full shard read rate without competing with the others. Which consumer model fits?

Enhanced fan-out. Each registered consumer gets up to 2 MB/s per shard through SubscribeToShard over HTTP/2. Standard and Provisioned modes allow 20 consumers; On-demand Advantage allows 50 in supported Regions.

How does KCL save Kinesis processing progress, and what happens if a worker crashes after processing but before checkpointing?

KCL stores the checkpoint sequence number in a DynamoDB lease table. The replacement resumes from the last checkpoint, so completed processing can repeat. Make side effects idempotent.

How does Kinesis Data Streams choose a shard, and how should a producer preserve strict write order?

By default it hashes the partition key with MD5 into a shard hash range. For strict ordering, make serial PutRecord calls to the same shard with SequenceNumberForOrdering. There is no stream-wide shard order.

A Kinesis producer sends many tiny application records and reaches the shard record-rate limit before its byte limit. Which KPL feature helps?

Aggregation. It packs multiple application records into one Kinesis record, reducing the number of service records written. Collection also reduces request overhead by using PutRecords.

Does Kinesis PutRecords guarantee strict ordering when some records in the batch fail?

No. PutRecords can partially succeed and does not guarantee ordering. Use serial PutRecord writes to the same shard with SequenceNumberForOrdering when strict ordering is required.

A KCL application has three metadata tables. Does every KCL 3.5 application always require all three?

No. KCL 3.5 supports consolidating metadata into one lease table. New migrations from 2.x use the single table format by default; existing 3.x applications must migrate.

A Kinesis producer retries PutRecord after a network timeout. Can duplicate payloads share one sequence number for deduplication?

No. If both writes succeeded, the payloads have different sequence numbers. Put an application-level identifier in the payload when duplicates must be detected.

A consumer uses enhanced fan-out and then restarts after a failed deployment. Does dedicated throughput guarantee exactly-once side effects?

No. Enhanced fan-out isolates read throughput. Restart from the last checkpoint can still repeat processing, so side effects must be idempotent.

Does the typical 70 ms enhanced fan-out propagation figure describe a guaranteed per-record latency?

No. AWS describes 70 ms as a typical average propagation delay.

Which KCL metric shows how far a Kinesis consumer application lags behind the stream, to scale the consumer fleet?

MillisBehindLatest.

Você concluiu todos os tópicos da DOP-C02!

Seu badge ficou dourado. Próximo passo: agendar a prova na AWS. Depois de passar, adicione-se ao Quem passou.

Compartilhar

Conte que você concluiu esta certificação

Concluí todos os tópicos da AWS Certified DevOps Engineer - Professional (DOP-C02) no Easy Review. Venha estudar também!