Easy ReviewSAA-C03
Design High-Performing Architectures11

Firehose

Deliver streaming records with destination buffers, optional transformation, and S3 recovery paths.

Firehose collects records, optionally transforms or converts them, and delivers them to a configured destination. Buffering, permissions, and failed-data handling determine how that delivery behaves.

Destinations and buffering

Amazon Data Firehose delivers streaming records to storage and analytics destinations. Its buffer collects records before delivery, so applications can send data without writing a destination delivery consumer.

How it works

  1. Send records through a supported source. Direct PUTDirect PUT sends application records to Firehose through its PutRecord or PutRecordBatch APIs. means application calls to Firehose through PutRecord or PutRecordBatch. CloudWatch Logs can use a subscription filter, which forwards selected log events. WAF can also send request logs to Firehose.

  2. For automatic subscription setup on new log groups, an EventBridge rule can match the CloudTrail-backed CreateLogGroup API event and invoke Lambda. The function calls PutSubscriptionFilter for the group, Firehose destination, and delivery roleA delivery role grants Firehose the permissions it needs to read sources, process records, and write destinations.. Its caller permissions need iam:PassRole.

  3. Give Firehose an IAM delivery role, a set of permissions it can assume. Its trust policyA trust policy specifies who can assume an IAM role. permits firehose.amazonaws.com; its permissions cover source reads and destination writes.

  4. If the source is Kinesis Data Streams, write to that source stream. Firehose starts reading at LATEST, the newest position, and disables its own PutRecord and PutRecordBatch ingestion calls.

  5. Firehose delivers when the size or interval buffer condition is met. S3 receives objects; OpenSearch receives bulk indexing requests. Redshift receives data through an intermediate S3 bucket followed by COPY, the database command that loads those objects.

Destination Buffer sizeA buffering hint tells Firehose how much data or time to accumulate before destination delivery. hint, default Interval hint, default
S3 1 to 128 MB, 5 MB 0 to 900 seconds, 300 seconds
OpenSearch 1 to 100 MB, 5 MB 0 to 900 seconds, 300 seconds
HTTP endpoint 1 to 64 MB, 5 MB 0 to 900 seconds, 60 seconds

Exam cues

  • If the question says ship CloudWatch Logs to S3 without a custom delivery consumer, think subscription filter to Firehose, because Firehose handles buffered delivery.
  • If the question says Firehose loads Redshift, think S3 then COPY, because Firehose stages the objects before loading them.
  • If the question says new log groups need automatic Firehose subscriptions, think EventBridge to Lambda calling PutSubscriptionFilter, because log-group creation can trigger subscription setup.
  • If the question says retained Kinesis history must be copied when Firehose is attached, think a separate replay consumer, because Firehose starts at LATEST.

Remember

  • S3 destination defaults: 5 MB or 300 seconds, whichever is met first.
  • S3 concatenation adds no delimiter by default. Enable the newline option when needed.
  • OpenSearch expects UTF-8 single-line JSON objects.
  • WAF request logs can use CloudWatch Logs, S3, or Firehose.

Trap: The buffering range belongs to the selected destination. Do not apply the S3 destination range to HTTP delivery or to S3 backup.

Firehose stages Redshift loads in S3
store batch1COPY2FirehoseiS3 stagingiRedshifti
  1. Firehose writes the buffered records to S3.

  2. Firehose issues COPY to load those S3 objects into Redshift.

Box details (3)
Firehose

Firehose buffers records before destination delivery.

S3 staging

Records are stored as S3 objects before the Redshift load.

Redshift

A COPY command loads the staged S3 objects into Redshift.

From the AWS docs (18)
  • Firehose delivers streaming records to a configured destination without a custom delivery consumer.
  • Firehose destination choices include S3, Redshift, OpenSearch Service, OpenSearch Serverless, Splunk, HTTP endpoints, Snowflake, and Apache Iceberg tables.
  • For S3 delivery, Firehose concatenates buffered records into objects without delimiters by default; a newline delimiter option is available.
  • For Redshift delivery, Firehose first writes records to S3 and then runs a Redshift COPY command to load them.
  • Firehose indexes buffered records in OpenSearch with bulk requests; input must be UTF-8 single-line JSON objects.
  • The first configured destination buffer condition met, size or interval, triggers Firehose delivery.
  • S3 destination buffer hints range from 1 to 128 MB, default 5 MB, and from 0 to 900 seconds, default 300 seconds.
  • OpenSearch destination buffer hints range from 1 to 100 MB, default 5 MB, and from 0 to 900 seconds, default 300 seconds.
  • HTTP endpoint destination buffer hints range from 1 to 64 MB, default 5 MB, and from 0 to 900 seconds, default 60 seconds.
  • When Kinesis Data Streams is the Firehose source, Firehose starts at LATEST and disables its own PutRecord and PutRecordBatch ingestion operations.
  • The Firehose delivery role grants required source read and destination write permissions; its trust policy allows firehose.amazonaws.com to assume it.
  • CloudWatch Logs subscription filters can forward log events to Firehose.
  • AWS WAF request logs can be sent to CloudWatch Logs, S3, or a Firehose delivery stream.
  • Applications can send records directly to Firehose through PutRecord or PutRecordBatch.
  • PutSubscriptionFilter creates or updates a CloudWatch Logs subscription for a log group and can target a same-account Firehose delivery stream.
  • Creating a subscription filter targeting Firehose requires iam:PassRole, and the delivery role grants CloudWatch Logs access to the destination stream.
  • EventBridge patterns can match CloudWatch Logs API events delivered by CloudTrail using source aws.logs, detail eventSource logs.amazonaws.com, and eventName.
  • An EventBridge rule can invoke Lambda asynchronously as its target.

Transformations and format conversion

Firehose transformation changes record content before delivery. Format conversion changes JSON into ParquetParquet is a columnar file format that Firehose can produce from JSON before S3 delivery. or ORCORC is a columnar file format that Firehose can produce from JSON before S3 delivery. for S3. These columnar formatsA columnar format stores data by column to reduce storage and query work. group values by column to reduce storage and query work. Compression is a separate destination setting that reduces the stored representation.

How it works

  1. Firehose separates aggregated records, meaning records combined by a producer. When configured, Kinesis Producer Library deaggregationDeaggregation separates multiple combined input records before further processing. runs before JSON or delimiter deaggregation.
  2. Firehose buffers the source records and invokes Lambda synchronously, meaning it waits for the response. Lambda can enrich records or change CSV input into JSON.
  3. If configured, partition data into S3 destination prefixes, then convert JSON to Parquet or ORC. Conversion uses a deserializerA deserializer reads encoded input data into the fields used by format conversion. to read JSON, a schemaA schema describes the field names and types used to interpret input records. describing fields and types in the Glue Data Catalog, and a serializerA serializer writes interpreted data into the chosen output format. to write the output format.
  4. Firehose applies destination buffering before S3 delivery. Configure S3 compression separately and enable source record backup when the original input must also be kept.

Exam cues

  • If the question says enrich records in an existing Firehose pipeline, think Lambda transformation, because Firehose invokes it before delivery.
  • If the question says CSV input must become Parquet, think Lambda to JSON followed by format conversion, because the built-in converter reads JSON.
  • If the question says output fields disappeared after conversion, think Glue schema, because output excludes fields absent from that schema.
  • If the question says compress S3 output without changing fields, think the S3 compression setting, because that feature does not require a Lambda transformation.

Remember

  • Transformation Lambda execution: up to 5 minutes.
  • Synchronous transformation payload: up to 6 MB for each request and response.
  • Lambda transformation buffer sizeA buffering hint tells Firehose how much data or time to accumulate before destination delivery. hint: 0.2 to 3 MB.
  • Conversion order: Lambda, data partitioning, format conversion, S3 delivery.
  • S3 compression options include GZIP, Snappy, Zip, Hadoop-Compatible Snappy, and none.

Trap: Compression does not require a Lambda transformation. Configure it separately in the S3 destination settings.

Transform content before S3 format conversion
invoke1return2deliver3Firehose inputiLambdaiFirehose outputiS3 objectsi
  1. Firehose invokes Lambda with buffered source records.

  2. Lambda returns transformed records; Firehose partitions and converts them when configured.

  3. Firehose delivers the converted output to S3.

Box details (4)
Firehose input

Configured deaggregation separates combined records before Lambda transformation.

Lambda

Transform records and return the original recordId, a result, and encoded data.

Firehose output

Configured partitioning precedes JSON conversion using the Glue schema.

S3 objects

Firehose delivers the converted records after destination buffering.

From the AWS docs (12)
  • Firehose buffers source records, invokes Lambda synchronously, receives the transformed records, then applies destination buffering before delivery.
  • Firehose transformation supports Lambda execution up to 5 minutes.
  • The Lambda transformation buffer size hint is configurable from 0.2 to 3 MB.
  • Firehose can convert JSON to Parquet or ORC before S3 delivery; non-JSON input can first be transformed to JSON with Lambda.
  • Parquet and ORC are columnar formats that organize data by columns and reduce storage and query work compared with row formats.
  • Firehose format conversion requires a JSON deserializer, a matching AWS Glue Data Catalog schema, and a Parquet or ORC serializer.
  • Fields absent from the Glue schema are absent from converted output; nested JSON requires matching STRUCT definitions.
  • Firehose S3 processing order is KPL deaggregation, JSON or delimiter deaggregation, Lambda processing, data partitioning, format conversion, then S3 delivery.
  • S3 compression is a separate Firehose destination setting with GZIP, Snappy, Zip, Hadoop-Compatible Snappy, or no compression options.
  • For supported sources, Firehose can back up source records to S3 when transformation or format conversion is configured for S3 delivery.
  • Athena can query data stored in S3 using SQL.
  • Each Firehose synchronous Lambda transformation request and response has a 6 MB payload limit.

Flashcards

tap to flip
Answer

·

All 11 cards as a list

Does a Firehose S3 destination deliver every input record immediately?

No. Firehose buffers records until its size or interval hint is met. S3 defaults are 5 MB and 300 seconds; ranges are 1 to 128 MB and 0 to 900 seconds.

Which Firehose destinations are available, and how does its Redshift destination load data?

Destinations include S3, Redshift, OpenSearch, Splunk, HTTP endpoints, Snowflake, and Iceberg tables. For Redshift, Firehose stages objects in S3 and then runs COPY.

A team connects Firehose to a Kinesis data stream that already has retained records. Will Firehose copy the historical records?

No. Firehose begins reading at LATEST. Use a consumer with an appropriate historical starting position if retained records must be replayed.

Firehose cannot read a Kinesis source or write S3 objects. Which role should the team inspect?

The Firehose delivery role. It needs source read permissions, destination permissions, and a trust policy allowing firehose.amazonaws.com to assume it.

A team wants managed delivery of CloudWatch Logs events to S3. Which path fits?

Use a CloudWatch Logs subscription filter targeting Firehose, with S3 as the Firehose destination. Firehose buffers records into S3 objects.

New CloudWatch log groups must automatically forward events through Firehose. How can the team automate subscription creation?

Match the CloudTrail-backed CreateLogGroup API event in EventBridge and invoke Lambda. Have the function call PutSubscriptionFilter for the new group with the Firehose destination and delivery role; the caller needs iam:PassRole.

Logs already flow through Firehose to S3 and need extra fields before delivery. How can the team keep the existing pipeline?

Configure a Lambda transformation on the Firehose stream. Firehose invokes the function synchronously before destination delivery.

How can Firehose reshape records before S3 or OpenSearch delivery and produce Parquet for an S3 destination?

Use a Lambda transformation to reshape the records. For S3, separately configure JSON-to-Parquet conversion with a matching Glue Data Catalog schema.

Network logs need enrichment before an S3 archive is queried by Athena. Which Firehose feature fits?

Use a Firehose Lambda transformation before S3 delivery. Athena can query the stored S3 data.

CSV records arrive in Firehose and must become Parquet in S3. What processing order should the team use?

Transform CSV to JSON with Lambda first, then enable JSON-to-Parquet conversion with a matching Glue schema. The built-in converter expects JSON.

A team only needs compressed S3 output from Firehose, without changing record fields. Must it deploy a Lambda transformation?

No. Configure S3 compression in the destination settings. Compression is separate from Lambda transformation.

You finished every topic of SAA-C03!

Your badge turned gold. Next step: book the exam on AWS. Once you pass, add yourself to Who passed.

Share

Share that you finished this certification

I finished every topic of AWS Certified Solutions Architect - Associate (SAA-C03) on Easy Review. Come study too!