Kinesis Data Streams divides retained records into shards. Producers choose partition keys, consumers choose read positions, and checkpoints let applications resume processing after interruption.
Shards, partition keys, and capacity
Kinesis Data Streams stores records for applications to process independently. A shardA shard is an ordered record sequence that provides a unit of stream capacity. is an ordered lane of records with its own capacity. The partition keyA partition key groups records by mapping them to a shard through a hash value. is the producer’s grouping value that normally decides which lane receives a record.
How it works
- A producer sends record bytes with a partition key. Kinesis hashes that key using MD5An MD5 hash maps a partition key to a number used to choose a shard hash range., a function that maps it to a number, and chooses the shard whose hash range contains it.
- Distribute traffic across keys. A hot keyA hot key concentrates traffic on one shard until the shard cannot accept more work., meaning one key carrying too much traffic, can throttle its shard even while other shards have room.
- Choose capacity management. Provisioned mode requires a shard count; on-demand manages shards automatically. On-demand can still throttle on a hot key or a sudden increase beyond its scaling envelope.
- Choose a maximum record size that downstream consumers can handle. Protect stored data with KMS server-side encryptionServer-side encryption protects record data stored in Kinesis using AWS KMS keys. and API connections with Transport Layer SecurityTLS encrypts network connections used to access Kinesis APIs., or TLS, which encrypts data in transit.
| Mode | Capacity responsibility | Cue |
|---|---|---|
| Provisioned | Choose shard count; split or merge shards | Manage known capacity explicitly |
| On-demand Standard | Kinesis manages shards | Variable, unpredictable traffic |
| On-demand Advantage | Account-level setting for on-demand streams | Proactively warm write capacity |
Exam cues
- If the question says most shards are idle but writes throttle, think hot partition key, because the same key maps to one shard.
- If the question says unpredictable traffic with no shard management, think On-demand Standard, because it automatically manages capacity.
- If the question says server log lines must reach a stream without changing the application, think Kinesis Agent, because it watches log files and sends new lines to the stream.
Remember
- Sustained shard writes: 1 MB/s or 1,000 service records/s. Shared reads: 2 MB/s and 5
GetRecordscalls/s. - New on-demand stream: 4 MB/s writes and 8 MB/s reads.
- On-demand supports twice the prior 30-day peak; exceeding that within 15 minutes can throttle.
- October 2025: AWS introduced records up to 10 MiBA mebibyte is a unit equal to 1,048,576 bytes.. The default maximum remains 1 MiB;
UpdateMaxRecordSizeaccepts 1024 to 10240 KiB. - Large records use intermittent burst capacityBurst capacity temporarily accepts intermittent large records above the sustained stream rate.. Check Region support and downstream limits, including Firehose destination restrictions.
- API connections require TLS 1.2; AWS recommends TLS 1.3.
Trap: A larger allowed record does not create a higher sustained shard write rate. Separate the per-record size setting from sustained capacity.
A partition key hash selects shard A when it falls in that range.
A hash in another range selects shard B; one hot key stays on its mapped shard.
Detalhes das caixas (3)
- Producer
Each record carries a partition key. Its MD5 hash maps it to a shard.
- Shard A
Records whose key hashes into this range use this shard and its capacity.
- Shard B
A different hash range can use a different shard. There is no order across shards.
Da documentação da AWS (20)
- A Kinesis data stream contains shards; a record contains a partition key, assigned sequence number, and data bytes.
- Kinesis uses an MD5 hash of the partition key to map records to a shard hash range; the same key maps to the same shard unless explicitly overridden.
- A shard supports sustained writes up to 1 MB per second; intermittent large records use burst capacity.
- Shared GetRecords consumers collectively have 2 MB per second per shard and up to 5 read transactions per second.
- On-demand streams can still throttle when one partition key exceeds the shard limit; spreading traffic across keys avoids concentrating it on one shard.
- Provisioned mode requires a chosen shard count and charges by shard-hour.
- On-demand Standard automatically manages shard capacity and suits variable, unpredictable traffic.
- On-demand Advantage is an account-level setting for on-demand streams and supports proactively warming write capacity.
- New on-demand streams start with 4 MB per second write and 8 MB per second read throughput.
- On-demand accommodates up to twice the prior 30-day peak write throughput; increases beyond twice that peak within 15 minutes can throttle.
- The default maximum record size is 1 MiB; the stream setting can be increased up to 10 MiB in supported Regions.
- UpdateMaxRecordSize accepts MaxRecordSizeInKiB values from 1024 to 10240 KiB, equivalent to 1 to 10 MiB.
- AWS announced support for record sizes up to 10 MiB in October 2025.
- Kinesis Data Streams supports server-side encryption of stored data with AWS KMS keys.
- Kinesis API clients must support TLS 1.2; AWS recommends TLS 1.3 for connections.
- Increasing the Kinesis maximum record size requires checking downstream limits; Firehose supports large-record S3 delivery but not dynamic partitioning, HTTP, Snowflake, or Redshift delivery.
- A shard supports writes of up to 1000 Kinesis service records per second.
- Resharding splits or merges shards to change provisioned stream capacity.
- Amazon Kinesis Data Streams collects and processes large streams of data records in real time.
- Kinesis Agent is a Java application for Linux servers that watches log files and sends new lines to a Kinesis data stream.
Enhanced fan-out and checkpoints
Enhanced fan-outEnhanced fan-out pushes records to each registered consumer with dedicated read throughput per shard. gives each registered consumer its own shardA shard is an ordered record sequence that provides a unit of stream capacity. read throughput. Kinesis Client Library, or KCL, coordinates workers and stores read progress. These solve separate problems: read isolation controls capacity, while checkpointingA checkpoint stores the last completed record position so a consumer can resume processing. records completed work.
How it works
- Choose shared reads when applications can share the shard’s 2 MB/s through
GetRecordspolling, meaning repeated requests for new records. Choose enhanced fan-out when each application needs dedicated read throughput. - Register the enhanced fan-out consumer with
RegisterStreamConsumer.SubscribeToShardpushes records over HTTP/2 with up to 2 MB/s per consumer per shard. - If using KCL, let its workers coordinate shard ownership through a leaseA lease records which KCL worker currently owns processing of a shard., the stored assignment of a shard to a worker. The DynamoDB lease table also stores each shard’s checkpoint, its saved completed position.
- Complete the processing side effect before checkpointing that progress. If the worker crashes between those operations, its replacement starts from the last checkpoint and can repeat the side effect.
- Make processing idempotent so a repeated record has no unintended extra effect.
Exam cues
- If the question says readers compete for the same shard throughput, think enhanced fan-out, because each registered consumer has independent capacity.
- If the question says processing completed before a worker crashed but checkpointing did not, think repeated processing after restart, because the saved position still points behind the completed work.
Remember
- Shared reads: 2 MB/s total per shard. Enhanced fan-out: up to 2 MB/s per consumer per shard.
- Standard and Provisioned: up to 20 registered enhanced fan-out consumers per stream.
- November 2025: AWS announced up to 50 for On-demand Advantage in supported Regions.
Trap: Enhanced fan-out does not make processing exactly once. Dedicated read capacity and saved checkpoints still require duplicate-safe side effects.
Consumer A receives its own enhanced fan-out stream of records.
Consumer B independently receives the same records.
Application A saves completed progress in its own lease metadata.
Application B saves its own progress independently.
Detalhes das caixas (4)
- Kinesis stream
Each enhanced fan-out consumer has dedicated throughput up to 2 MB/s per shard.
- KCL app A
This application independently processes the stream and checkpoints its progress.
- KCL app B
Another application can process the same records independently.
- Lease metadata
Each KCL application maintains its own DynamoDB lease table with shard ownership and checkpoints.
Da documentação da AWS (10)
- Shared-throughput consumers poll GetRecords and share 2 MB per second of read throughput per shard.
- An enhanced fan-out consumer receives dedicated throughput up to 2 MB per second per shard, independent of other consumers.
- Enhanced fan-out pushes records over HTTP/2 with SubscribeToShard instead of GetRecords polling.
- Enhanced fan-out consumers are registered for a stream using RegisterStreamConsumer and subscribe to shards with SubscribeToShard.
- On-demand Standard and Provisioned streams support up to 20 registered enhanced fan-out consumers per stream; On-demand Advantage supports up to 50 in supported Regions.
- AWS announced support for 50 enhanced fan-out consumers on On-demand Advantage streams in November 2025.
- KCL uses a DynamoDB lease table to track which worker processes a shard and the most recent checkpoint sequence number.
- When a KCL processor restarts after worker failure, deployment, or resharding, it resumes from the last checkpoint and can repeat completed processing.
- A checkpoint records completed progress; a crash after a side effect but before checkpointing can cause that side effect to be attempted again.
- Consumers should handle records more than once without unintended extra side effects; this is idempotent processing.
Flashcards
Nada para revisar aqui agora. Escolha outros tópicos ou outro filtro.
·
Todos os 10 flashcards em lista
What sustained write and shared-read capacity does one Kinesis Data Streams shard provide?
Writes: 1 MB/s or 1000 records/s. Shared reads: 2 MB/s with up to 5 GetRecords calls/s. Large-record burst support does not raise the sustained rate.
A Kinesis Data Streams stream throttles writes while most shards are idle. What should the team investigate?
A hot partition key can overload one shard. Distribute records across more keys; on-demand capacity or extra shards does not remove a single key hotspot.
How do Kinesis Data Streams Provisioned and On-demand Standard capacity modes differ?
Provisioned requires choosing and managing shard capacity and charges by shard-hour. On-demand Standard manages shards automatically and suits unpredictable traffic.
A Kinesis application needs occasional 4 MiB records. Is the default stream setting enough, and does enabling large records raise sustained shard throughput?
Raise the maximum record-size setting from its 1 MiB default, up to 10 MiB in supported Regions. Large records use burst capacity; sustained shard limits remain unchanged. Check downstream compatibility.
An on-demand Kinesis stream receives more than twice its recent peak within 15 minutes and throttles. Can on-demand scaling eliminate this case?
No. On-demand accommodates up to twice the previous 30-day peak; faster increases can throttle and require retries.
A company needs to collect new log lines from Linux servers for real-time stream processing. Which Amazon Kinesis option fits?
Use Kinesis Agent to send new log lines to Amazon Kinesis Data Streams for real-time processing.
A company must ingest and process website clickstream data in real time. Which AWS service fits?
Amazon Kinesis Data Streams.
Web servers write log files, and the lines must reach a Kinesis data stream without changing the application. What tool does that?
Kinesis Agent, installed on the servers and configured with the files to watch and the target stream.
Five Kinesis Data Streams applications each need the full shard read rate without competing with the others. Which consumer model fits?
Enhanced fan-out. Each registered consumer gets up to 2 MB/s per shard through SubscribeToShard over HTTP/2. Standard and Provisioned modes allow 20 consumers; On-demand Advantage allows 50 in supported Regions.
How does KCL save Kinesis processing progress, and what happens if a worker crashes after processing but before checkpointing?
KCL stores the checkpoint sequence number in a DynamoDB lease table. The replacement resumes from the last checkpoint, so completed processing can repeat. Make side effects idempotent.