Easy ReviewSAA-C03
Design High-Performing Architectures10

Kinesis Data Streams

Retain streaming records for independent consumers, with explicit capacity, ordering, and replay choices.

Kinesis Data Streams divides retained records into shards. Producers choose partition keys, consumers choose read positions, and checkpoints let applications resume processing after interruption.

Shards, partition keys, and capacity

Kinesis Data Streams stores records for applications to process independently. A shardA shard is an ordered record sequence that provides a unit of stream capacity. is an ordered lane of records with its own capacity. The partition keyA partition key groups records by mapping them to a shard through a hash value. is the producer’s grouping value that normally decides which lane receives a record.

How it works

  1. A producer sends record bytes with a partition key. Kinesis hashes that key using MD5An MD5 hash maps a partition key to a number used to choose a shard hash range., a function that maps it to a number, and chooses the shard whose hash range contains it.
  2. Distribute traffic across keys. A hot keyA hot key concentrates traffic on one shard until the shard cannot accept more work., meaning one key carrying too much traffic, can throttle its shard even while other shards have room.
  3. Choose capacity management. Provisioned mode requires a shard count; on-demand manages shards automatically. On-demand can still throttle on a hot key or a sudden increase beyond its scaling envelope.
  4. Choose a maximum record size that downstream consumers can handle. Protect stored data with KMS server-side encryptionServer-side encryption protects record data stored in Kinesis using AWS KMS keys. and API connections with Transport Layer SecurityTLS encrypts network connections used to access Kinesis APIs., or TLS, which encrypts data in transit.
Mode Capacity responsibility Cue
Provisioned Choose shard count; split or merge shards Manage known capacity explicitly
On-demand Standard Kinesis manages shards Variable, unpredictable traffic
On-demand Advantage Account-level setting for on-demand streams Proactively warm write capacity

Exam cues

  • If the question says most shards are idle but writes throttle, think hot partition key, because the same key maps to one shard.
  • If the question says unpredictable traffic with no shard management, think On-demand Standard, because it automatically manages capacity.
  • If the question says server log lines must reach a stream without changing the application, think Kinesis Agent, because it watches log files and sends new lines to the stream.

Remember

  • Sustained shard writes: 1 MB/s or 1,000 service records/s. Shared reads: 2 MB/s and 5 GetRecords calls/s.
  • New on-demand stream: 4 MB/s writes and 8 MB/s reads.
  • On-demand supports twice the prior 30-day peak; exceeding that within 15 minutes can throttle.
  • October 2025: AWS introduced records up to 10 MiBA mebibyte is a unit equal to 1,048,576 bytes.. The default maximum remains 1 MiB; UpdateMaxRecordSize accepts 1024 to 10240 KiB.
  • Large records use intermittent burst capacityBurst capacity temporarily accepts intermittent large records above the sustained stream rate.. Check Region support and downstream limits, including Firehose destination restrictions.
  • API connections require TLS 1.2; AWS recommends TLS 1.3.

Trap: A larger allowed record does not create a higher sustained shard write rate. Separate the per-record size setting from sustained capacity.

Partition keys distribute records among shards
hash range A1hash range B2ProduceriShard AiShard Bi
  1. A partition key hash selects shard A when it falls in that range.

  2. A hash in another range selects shard B; one hot key stays on its mapped shard.

Box details (3)
Producer

Each record carries a partition key. Its MD5 hash maps it to a shard.

Shard A

Records whose key hashes into this range use this shard and its capacity.

Shard B

A different hash range can use a different shard. There is no order across shards.

From the AWS docs (20)
  • A Kinesis data stream contains shards; a record contains a partition key, assigned sequence number, and data bytes.
  • Kinesis uses an MD5 hash of the partition key to map records to a shard hash range; the same key maps to the same shard unless explicitly overridden.
  • A shard supports sustained writes up to 1 MB per second; intermittent large records use burst capacity.
  • Shared GetRecords consumers collectively have 2 MB per second per shard and up to 5 read transactions per second.
  • On-demand streams can still throttle when one partition key exceeds the shard limit; spreading traffic across keys avoids concentrating it on one shard.
  • Provisioned mode requires a chosen shard count and charges by shard-hour.
  • On-demand Standard automatically manages shard capacity and suits variable, unpredictable traffic.
  • On-demand Advantage is an account-level setting for on-demand streams and supports proactively warming write capacity.
  • New on-demand streams start with 4 MB per second write and 8 MB per second read throughput.
  • On-demand accommodates up to twice the prior 30-day peak write throughput; increases beyond twice that peak within 15 minutes can throttle.
  • The default maximum record size is 1 MiB; the stream setting can be increased up to 10 MiB in supported Regions.
  • UpdateMaxRecordSize accepts MaxRecordSizeInKiB values from 1024 to 10240 KiB, equivalent to 1 to 10 MiB.
  • AWS announced support for record sizes up to 10 MiB in October 2025.
  • Kinesis Data Streams supports server-side encryption of stored data with AWS KMS keys.
  • Kinesis API clients must support TLS 1.2; AWS recommends TLS 1.3 for connections.
  • Increasing the Kinesis maximum record size requires checking downstream limits; Firehose supports large-record S3 delivery but not dynamic partitioning, HTTP, Snowflake, or Redshift delivery.
  • A shard supports writes of up to 1000 Kinesis service records per second.
  • Resharding splits or merges shards to change provisioned stream capacity.
  • Amazon Kinesis Data Streams collects and processes large streams of data records in real time.
  • Kinesis Agent is a Java application for Linux servers that watches log files and sends new lines to a Kinesis data stream.

Enhanced fan-out and checkpoints

Enhanced fan-outEnhanced fan-out pushes records to each registered consumer with dedicated read throughput per shard. gives each registered consumer its own shardA shard is an ordered record sequence that provides a unit of stream capacity. read throughput. Kinesis Client Library, or KCL, coordinates workers and stores read progress. These solve separate problems: read isolation controls capacity, while checkpointingA checkpoint stores the last completed record position so a consumer can resume processing. records completed work.

How it works

  1. Choose shared reads when applications can share the shard’s 2 MB/s through GetRecords polling, meaning repeated requests for new records. Choose enhanced fan-out when each application needs dedicated read throughput.
  2. Register the enhanced fan-out consumer with RegisterStreamConsumer. SubscribeToShard pushes records over HTTP/2 with up to 2 MB/s per consumer per shard.
  3. If using KCL, let its workers coordinate shard ownership through a leaseA lease records which KCL worker currently owns processing of a shard., the stored assignment of a shard to a worker. The DynamoDB lease table also stores each shard’s checkpoint, its saved completed position.
  4. Complete the processing side effect before checkpointing that progress. If the worker crashes between those operations, its replacement starts from the last checkpoint and can repeat the side effect.
  5. Make processing idempotent so a repeated record has no unintended extra effect.

Exam cues

  • If the question says readers compete for the same shard throughput, think enhanced fan-out, because each registered consumer has independent capacity.
  • If the question says processing completed before a worker crashed but checkpointing did not, think repeated processing after restart, because the saved position still points behind the completed work.

Remember

  • Shared reads: 2 MB/s total per shard. Enhanced fan-out: up to 2 MB/s per consumer per shard.
  • Standard and Provisioned: up to 20 registered enhanced fan-out consumers per stream.
  • November 2025: AWS announced up to 50 for On-demand Advantage in supported Regions.

Trap: Enhanced fan-out does not make processing exactly once. Dedicated read capacity and saved checkpoints still require duplicate-safe side effects.

Independent consumers keep separate read progress
push1push2checkpoint A3checkpoint B4Kinesis streamiKCL app AiKCL app BiLease metadatai
  1. Consumer A receives its own enhanced fan-out stream of records.

  2. Consumer B independently receives the same records.

  3. Application A saves completed progress in its own lease metadata.

  4. Application B saves its own progress independently.

Box details (4)
Kinesis stream

Each enhanced fan-out consumer has dedicated throughput up to 2 MB/s per shard.

KCL app A

This application independently processes the stream and checkpoints its progress.

KCL app B

Another application can process the same records independently.

Lease metadata

Each KCL application maintains its own DynamoDB lease table with shard ownership and checkpoints.

From the AWS docs (10)
  • Shared-throughput consumers poll GetRecords and share 2 MB per second of read throughput per shard.
  • An enhanced fan-out consumer receives dedicated throughput up to 2 MB per second per shard, independent of other consumers.
  • Enhanced fan-out pushes records over HTTP/2 with SubscribeToShard instead of GetRecords polling.
  • Enhanced fan-out consumers are registered for a stream using RegisterStreamConsumer and subscribe to shards with SubscribeToShard.
  • On-demand Standard and Provisioned streams support up to 20 registered enhanced fan-out consumers per stream; On-demand Advantage supports up to 50 in supported Regions.
  • AWS announced support for 50 enhanced fan-out consumers on On-demand Advantage streams in November 2025.
  • KCL uses a DynamoDB lease table to track which worker processes a shard and the most recent checkpoint sequence number.
  • When a KCL processor restarts after worker failure, deployment, or resharding, it resumes from the last checkpoint and can repeat completed processing.
  • A checkpoint records completed progress; a crash after a side effect but before checkpointing can cause that side effect to be attempted again.
  • Consumers should handle records more than once without unintended extra side effects; this is idempotent processing.

Flashcards

tap to flip
Answer

·

All 10 cards as a list

What sustained write and shared-read capacity does one Kinesis Data Streams shard provide?

Writes: 1 MB/s or 1000 records/s. Shared reads: 2 MB/s with up to 5 GetRecords calls/s. Large-record burst support does not raise the sustained rate.

A Kinesis Data Streams stream throttles writes while most shards are idle. What should the team investigate?

A hot partition key can overload one shard. Distribute records across more keys; on-demand capacity or extra shards does not remove a single key hotspot.

How do Kinesis Data Streams Provisioned and On-demand Standard capacity modes differ?

Provisioned requires choosing and managing shard capacity and charges by shard-hour. On-demand Standard manages shards automatically and suits unpredictable traffic.

A Kinesis application needs occasional 4 MiB records. Is the default stream setting enough, and does enabling large records raise sustained shard throughput?

Raise the maximum record-size setting from its 1 MiB default, up to 10 MiB in supported Regions. Large records use burst capacity; sustained shard limits remain unchanged. Check downstream compatibility.

An on-demand Kinesis stream receives more than twice its recent peak within 15 minutes and throttles. Can on-demand scaling eliminate this case?

No. On-demand accommodates up to twice the previous 30-day peak; faster increases can throttle and require retries.

A company needs to collect new log lines from Linux servers for real-time stream processing. Which Amazon Kinesis option fits?

Use Kinesis Agent to send new log lines to Amazon Kinesis Data Streams for real-time processing.

A company must ingest and process website clickstream data in real time. Which AWS service fits?

Amazon Kinesis Data Streams.

Web servers write log files, and the lines must reach a Kinesis data stream without changing the application. What tool does that?

Kinesis Agent, installed on the servers and configured with the files to watch and the target stream.

Five Kinesis Data Streams applications each need the full shard read rate without competing with the others. Which consumer model fits?

Enhanced fan-out. Each registered consumer gets up to 2 MB/s per shard through SubscribeToShard over HTTP/2. Standard and Provisioned modes allow 20 consumers; On-demand Advantage allows 50 in supported Regions.

How does KCL save Kinesis processing progress, and what happens if a worker crashes after processing but before checkpointing?

KCL stores the checkpoint sequence number in a DynamoDB lease table. The replacement resumes from the last checkpoint, so completed processing can repeat. Make side effects idempotent.

You finished every topic of SAA-C03!

Your badge turned gold. Next step: book the exam on AWS. Once you pass, add yourself to Who passed.

Share

Share that you finished this certification

I finished every topic of AWS Certified Solutions Architect - Associate (SAA-C03) on Easy Review. Come study too!