Firehose collects records, optionally transforms or converts them, and delivers them to a configured destination. Buffering, permissions, and failed-data handling determine how that delivery behaves.
Destinations and buffering
Amazon Data Firehose delivers streaming records to storage and analytics destinations. Its buffer collects records before delivery, so applications can send data without writing a destination delivery consumer.
How it works
-
Send records through a supported source. Direct PUTDirect PUT sends application records to Firehose through its PutRecord or PutRecordBatch APIs. means application calls to Firehose through
PutRecordorPutRecordBatch. CloudWatch Logs can use a subscription filter, which forwards selected log events. WAF can also send request logs to Firehose. -
For automatic subscription setup on new log groups, an EventBridge rule can match the CloudTrail-backed
CreateLogGroupAPI event and invoke Lambda. The function callsPutSubscriptionFilterfor the group, Firehose destination, and delivery roleA delivery role grants Firehose the permissions it needs to read sources, process records, and write destinations.. Its caller permissions neediam:PassRole. -
Give Firehose an IAM delivery role, a set of permissions it can assume. Its trust policyA trust policy specifies who can assume an IAM role. permits
firehose.amazonaws.com; its permissions cover source reads and destination writes. -
If the source is Kinesis Data Streams, write to that source stream. Firehose starts reading at
LATEST, the newest position, and disables its ownPutRecordandPutRecordBatchingestion calls. -
Firehose delivers when the size or interval buffer condition is met. S3 receives objects; OpenSearch receives bulk indexing requests. Redshift receives data through an intermediate S3 bucket followed by
COPY, the database command that loads those objects.
| Destination | Buffer sizeA buffering hint tells Firehose how much data or time to accumulate before destination delivery. hint, default | Interval hint, default |
|---|---|---|
| S3 | 1 to 128 MB, 5 MB | 0 to 900 seconds, 300 seconds |
| OpenSearch | 1 to 100 MB, 5 MB | 0 to 900 seconds, 300 seconds |
| HTTP endpoint | 1 to 64 MB, 5 MB | 0 to 900 seconds, 60 seconds |
Exam cues
- If the question says ship CloudWatch Logs to S3 without a custom delivery consumer, think subscription filter to Firehose, because Firehose handles buffered delivery.
- If the question says Firehose loads Redshift, think S3 then
COPY, because Firehose stages the objects before loading them. - If the question says new log groups need automatic Firehose subscriptions, think EventBridge to Lambda calling
PutSubscriptionFilter, because log-group creation can trigger subscription setup. - If the question says retained Kinesis history must be copied when Firehose is attached, think a separate replay consumer, because Firehose starts at
LATEST.
Remember
- S3 destination defaults: 5 MB or 300 seconds, whichever is met first.
- S3 concatenation adds no delimiter by default. Enable the newline option when needed.
- OpenSearch expects UTF-8 single-line JSON objects.
- WAF request logs can use CloudWatch Logs, S3, or Firehose.
Trap: The buffering range belongs to the selected destination. Do not apply the S3 destination range to HTTP delivery or to S3 backup.
Firehose writes the buffered records to S3.
Firehose issues COPY to load those S3 objects into Redshift.
Detalhes das caixas (3)
- Firehose
Firehose buffers records before destination delivery.
- S3 staging
Records are stored as S3 objects before the Redshift load.
- Redshift
A COPY command loads the staged S3 objects into Redshift.
Da documentação da AWS (18)
- Firehose delivers streaming records to a configured destination without a custom delivery consumer.
- Firehose destination choices include S3, Redshift, OpenSearch Service, OpenSearch Serverless, Splunk, HTTP endpoints, Snowflake, and Apache Iceberg tables.
- For S3 delivery, Firehose concatenates buffered records into objects without delimiters by default; a newline delimiter option is available.
- For Redshift delivery, Firehose first writes records to S3 and then runs a Redshift COPY command to load them.
- Firehose indexes buffered records in OpenSearch with bulk requests; input must be UTF-8 single-line JSON objects.
- The first configured destination buffer condition met, size or interval, triggers Firehose delivery.
- S3 destination buffer hints range from 1 to 128 MB, default 5 MB, and from 0 to 900 seconds, default 300 seconds.
- OpenSearch destination buffer hints range from 1 to 100 MB, default 5 MB, and from 0 to 900 seconds, default 300 seconds.
- HTTP endpoint destination buffer hints range from 1 to 64 MB, default 5 MB, and from 0 to 900 seconds, default 60 seconds.
- When Kinesis Data Streams is the Firehose source, Firehose starts at LATEST and disables its own PutRecord and PutRecordBatch ingestion operations.
- The Firehose delivery role grants required source read and destination write permissions; its trust policy allows firehose.amazonaws.com to assume it.
- CloudWatch Logs subscription filters can forward log events to Firehose.
- AWS WAF request logs can be sent to CloudWatch Logs, S3, or a Firehose delivery stream.
- Applications can send records directly to Firehose through PutRecord or PutRecordBatch.
- PutSubscriptionFilter creates or updates a CloudWatch Logs subscription for a log group and can target a same-account Firehose delivery stream.
- Creating a subscription filter targeting Firehose requires iam:PassRole, and the delivery role grants CloudWatch Logs access to the destination stream.
- EventBridge patterns can match CloudWatch Logs API events delivered by CloudTrail using source aws.logs, detail eventSource logs.amazonaws.com, and eventName.
- An EventBridge rule can invoke Lambda asynchronously as its target.
Transformations and format conversion
Firehose transformation changes record content before delivery. Format conversion changes JSON into ParquetParquet is a columnar file format that Firehose can produce from JSON before S3 delivery. or ORCORC is a columnar file format that Firehose can produce from JSON before S3 delivery. for S3. These columnar formatsA columnar format stores data by column to reduce storage and query work. group values by column to reduce storage and query work. Compression is a separate destination setting that reduces the stored representation.
How it works
- Firehose separates aggregated records, meaning records combined by a producer. When configured, Kinesis Producer Library deaggregationDeaggregation separates multiple combined input records before further processing. runs before JSON or delimiter deaggregation.
- Firehose buffers the source records and invokes Lambda synchronously, meaning it waits for the response. Lambda can enrich records or change CSV input into JSON.
- If configured, partition data into S3 destination prefixes, then convert JSON to Parquet or ORC. Conversion uses a deserializerA deserializer reads encoded input data into the fields used by format conversion. to read JSON, a schemaA schema describes the field names and types used to interpret input records. describing fields and types in the Glue Data Catalog, and a serializerA serializer writes interpreted data into the chosen output format. to write the output format.
- Firehose applies destination buffering before S3 delivery. Configure S3 compression separately and enable source record backup when the original input must also be kept.
Exam cues
- If the question says enrich records in an existing Firehose pipeline, think Lambda transformation, because Firehose invokes it before delivery.
- If the question says CSV input must become Parquet, think Lambda to JSON followed by format conversion, because the built-in converter reads JSON.
- If the question says output fields disappeared after conversion, think Glue schema, because output excludes fields absent from that schema.
- If the question says compress S3 output without changing fields, think the S3 compression setting, because that feature does not require a Lambda transformation.
Remember
- Transformation Lambda execution: up to 5 minutes.
- Synchronous transformation payload: up to 6 MB for each request and response.
- Lambda transformation buffer sizeA buffering hint tells Firehose how much data or time to accumulate before destination delivery. hint: 0.2 to 3 MB.
- Conversion order: Lambda, data partitioning, format conversion, S3 delivery.
- S3 compression options include GZIP, Snappy, Zip, Hadoop-Compatible Snappy, and none.
Trap: Compression does not require a Lambda transformation. Configure it separately in the S3 destination settings.
Firehose invokes Lambda with buffered source records.
Lambda returns transformed records; Firehose partitions and converts them when configured.
Firehose delivers the converted output to S3.
Detalhes das caixas (4)
- Firehose input
Configured deaggregation separates combined records before Lambda transformation.
- Lambda
Transform records and return the original recordId, a result, and encoded data.
- Firehose output
Configured partitioning precedes JSON conversion using the Glue schema.
- S3 objects
Firehose delivers the converted records after destination buffering.
Da documentação da AWS (12)
- Firehose buffers source records, invokes Lambda synchronously, receives the transformed records, then applies destination buffering before delivery.
- Firehose transformation supports Lambda execution up to 5 minutes.
- The Lambda transformation buffer size hint is configurable from 0.2 to 3 MB.
- Firehose can convert JSON to Parquet or ORC before S3 delivery; non-JSON input can first be transformed to JSON with Lambda.
- Parquet and ORC are columnar formats that organize data by columns and reduce storage and query work compared with row formats.
- Firehose format conversion requires a JSON deserializer, a matching AWS Glue Data Catalog schema, and a Parquet or ORC serializer.
- Fields absent from the Glue schema are absent from converted output; nested JSON requires matching STRUCT definitions.
- Firehose S3 processing order is KPL deaggregation, JSON or delimiter deaggregation, Lambda processing, data partitioning, format conversion, then S3 delivery.
- S3 compression is a separate Firehose destination setting with GZIP, Snappy, Zip, Hadoop-Compatible Snappy, or no compression options.
- For supported sources, Firehose can back up source records to S3 when transformation or format conversion is configured for S3 delivery.
- Athena can query data stored in S3 using SQL.
- Each Firehose synchronous Lambda transformation request and response has a 6 MB payload limit.
Delivery failures and backup
Firehose recovery depends on the destination and source. Direct PUTDirect PUT sends application records to Firehose through its PutRecord or PutRecordBatch APIs. means applications send records through Firehose ingestion APIs. S3 backup preserves failed records for investigation and manual backfillBackfill loads data that an earlier delivery attempt skipped., meaning loading records that earlier delivery skipped. It must remain writable for that recovery path to work.
How it works
- Configure the S3 backup bucket and prefixes for a supported source and destination. With supported application destinations, choose all source records or failed data only.
- Firehose retries destination failures according to that destination’s policy. For OpenSearch, choose a retry duration from 0 to 7200 seconds.
- After OpenSearch retries expire, inspect skipped documents in
AmazonOpenSearchService_failed/. For Redshift, theerrors/manifestA manifest identifies the S3 objects skipped during a Redshift load so they can be loaded manually. lists skipped S3 objects for manual loading. A manifest is a file identifying the objects to load. - Inspect transformation failures separately. Network timeout or Lambda invocation-limit failures get three retries by default. Unsuccessfully transformed records go to
processing-failedwith error details and raw data. - Restore S3 backup access if delivery stalls. When both the primary destination and failed-data backup fail, Firehose pauses delivery. Format conversion errors also block delivery if their S3 error write cannot succeed.
Exam cues
- If the question says OpenSearch rejects documents, think S3 failed-document backup and manual backfill, because retry exhaustion sends skipped documents there.
- If the question says Firehose is stalled despite a backup bucket being configured, think backup write failures, because configuration alone does not ensure the bucket is writable.
- If the question says Redshift loads were skipped, think the
errors/manifest, because it identifies S3 objects for manual loading.
Remember
- S3 backup settings support Direct PUT and Kinesis Data Streams sources.
- DirectPut to S3: retries up to 24 hours. Kinesis or MSK to S3: retries are bounded by source retention.
- OpenSearch retry duration: 0 to 7200 seconds.
- Transformation invocation retry default for network timeouts or invocation limits: 3 retries.
- Firehose uses at-least-once deliveryAt-least-once delivery allows a record to be delivered again when an attempt is retried.. The documented retry-duplicate exception covers S3, Iceberg, and Snowflake destinations.
Trap: An S3 backup setting does not guarantee indefinite retention or automatic replay. Keep the backup writable and perform manual backfill for skipped OpenSearch documents or Redshift loads.
Da documentação da AWS (13)
- Firehose uses at-least-once delivery; retries can introduce duplicates except for S3, Apache Iceberg tables, and Snowflake destinations.
- Firehose backup settings support Direct PUT and Kinesis Data Streams sources; supported application destinations can back up all source data or failed data only to S3.
- Firehose backup configuration includes an S3 backup bucket, bucket prefix, and error output prefix for failed data.
- When both destination delivery and writing failed data to the S3 backup location fail, Firehose pauses delivery until one path succeeds.
- For an S3 destination with a DirectPut source, Firehose retries failed delivery for up to 24 hours; data can be lost after that storage window.
- For an S3 destination with Kinesis Data Streams or MSK sources, failed delivery retries are bounded by the source stream retention policy.
- OpenSearch and OpenSearch Serverless delivery retry duration is configurable from 0 to 7200 seconds.
- After OpenSearch retries expire, Firehose writes skipped documents to S3 under AmazonOpenSearchService_failed/ for manual backfill.
- After the configured Redshift retry duration expires, Firehose writes skipped S3 object information to a manifest in errors/ for manual backfill.
- Firehose retries Lambda invocations three times by default when network timeout or invocation limits cause failure, then treats the skipped batch as unsuccessfully processed.
- Firehose writes unsuccessfully transformed records to the S3 processing-failed folder with error details and the raw record.
- Format conversion parse errors write failed records to an S3 error prefix; if that write fails, Firehose retries it indefinitely and blocks further delivery.
- Direct PUT sends application records to Firehose through PutRecord or PutRecordBatch.
Logging and transformation details
Amazon Data Firehose can receive AWS Network Firewall logs and process records returned by Lambda. Destination settings, buffering restrictions, and the transformation response determine how these configurations work.
How it works
- For direct Network Firewall log delivery, set
LogDestinationTypetoKinesisDataFirehoseand put the stream name indeliveryStream. See the logging destination configuration. - For Direct PUTDirect PUT sends application records to Firehose through its PutRecord or PutRecordBatch APIs. or Kinesis Data Streams sources, each transformed Lambda output record must keep its original
recordIdand include aresultand base64-encodeddata. A mismatched ID causes transformation failure. See the transformation response requirements. - Firehose treats
Okas successfully transformed andDroppedas intentionally discarded. Both count as successful processing;ProcessingFailedindicates failure.
Exam cues
- If the question says Network Firewall flow logs must go directly to Firehose, think
KinesisDataFirehoseanddeliveryStream, because those settings select the stream. - If the question says a transformation returns a new
recordId, think transformation failure, because the output must preserve the original ID. - If the question says a Lambda response marks a record
Dropped, think successful processing, because the record was intentionally discarded. - If the question says dynamic partitioningDynamic partitioning uses record values to send data to selected S3 prefixes. needs zero buffering, think a different buffering setting, because zero buffering is unavailable for dynamic partitioning.
Remember
- Zero buffering is unavailable for S3 backup and dynamic partitioning.
- Lambda result values are
Ok,Dropped, andProcessingFailed.
Trap:
Droppedcounts as successful processing, but changing a record’srecordIdcauses transformation failure.
Da documentação da AWS (4)
- AWS Network Firewall supports direct log delivery to Firehose using LogDestinationType KinesisDataFirehose and the deliveryStream destination key.
- Zero buffering is unavailable for S3 backup and for dynamic partitioning.
- For Direct PUT and Kinesis Data Streams sources, every Lambda output record must preserve recordId and include result and base64-encoded data.
- The Lambda result values are Ok, Dropped, and ProcessingFailed; Firehose treats Ok and Dropped as successfully processed.
Flashcards
Nada para revisar aqui agora. Escolha outros tópicos ou outro filtro.
·
Todos os 21 flashcards em lista
Does a Firehose S3 destination deliver every input record immediately?
No. Firehose buffers records until its size or interval hint is met. S3 defaults are 5 MB and 300 seconds; ranges are 1 to 128 MB and 0 to 900 seconds.
Which Firehose destinations are available, and how does its Redshift destination load data?
Destinations include S3, Redshift, OpenSearch, Splunk, HTTP endpoints, Snowflake, and Iceberg tables. For Redshift, Firehose stages objects in S3 and then runs COPY.
A team connects Firehose to a Kinesis data stream that already has retained records. Will Firehose copy the historical records?
No. Firehose begins reading at LATEST. Use a consumer with an appropriate historical starting position if retained records must be replayed.
Firehose cannot read a Kinesis source or write S3 objects. Which role should the team inspect?
The Firehose delivery role. It needs source read permissions, destination permissions, and a trust policy allowing firehose.amazonaws.com to assume it.
A team wants managed delivery of CloudWatch Logs events to S3. Which path fits?
Use a CloudWatch Logs subscription filter targeting Firehose, with S3 as the Firehose destination. Firehose buffers records into S3 objects.
New CloudWatch log groups must automatically forward events through Firehose. How can the team automate subscription creation?
Match the CloudTrail-backed CreateLogGroup API event in EventBridge and invoke Lambda. Have the function call PutSubscriptionFilter for the new group with the Firehose destination and delivery role; the caller needs iam:PassRole.
Logs already flow through Firehose to S3 and need extra fields before delivery. How can the team keep the existing pipeline?
Configure a Lambda transformation on the Firehose stream. Firehose invokes the function synchronously before destination delivery.
How can Firehose reshape records before S3 or OpenSearch delivery and produce Parquet for an S3 destination?
Use a Lambda transformation to reshape the records. For S3, separately configure JSON-to-Parquet conversion with a matching Glue Data Catalog schema.
Network logs need enrichment before an S3 archive is queried by Athena. Which Firehose feature fits?
Use a Firehose Lambda transformation before S3 delivery. Athena can query the stored S3 data.
CSV records arrive in Firehose and must become Parquet in S3. What processing order should the team use?
Transform CSV to JSON with Lambda first, then enable JSON-to-Parquet conversion with a matching Glue schema. The built-in converter expects JSON.
A team only needs compressed S3 output from Firehose, without changing record fields. Must it deploy a Lambda transformation?
No. Configure S3 compression in the destination settings. Compression is separate from Lambda transformation.
Firehose cannot index some documents into OpenSearch. What should the team configure and how does it recover skipped documents?
Configure S3 backup for failed documents or all source data. After the retry duration expires, skipped documents go to AmazonOpenSearchService_failed/ for manual backfill. Backup settings support Direct PUT and Kinesis Data Streams sources.
Does Firehose retry behavior provide a general exactly-once guarantee for every destination?
No. Firehose documents at-least-once delivery. Retry duplicates apply except to S3, Iceberg tables, and Snowflake destinations; do not extend that exception to the whole pipeline.
A Firehose stream cannot deliver to its application destination and cannot write to its S3 error backup. What happens?
Firehose pauses delivery until destination delivery or the S3 failed-data write can succeed. Restore backup access as well as investigating the primary destination.
A Firehose Lambda invocation repeatedly fails because of invocation limits. What is the default retry count and where do failed records go?
Three retries by default. If the invocation still fails, Firehose treats the skipped batch as unsuccessful and writes the failed records with error details to S3 processing-failed.
What retry duration range can a Firehose OpenSearch destination use?
0 to 7200 seconds. After the duration expires, skipped documents are written to S3 for manual backfill.
A Firehose DirectPut stream has been unable to write to S3 for more than a day. Can its internal storage prevent indefinite data loss?
No. DirectPut retries to an S3 destination are limited to 24 hours. Stream-backed sources have their own retention boundary.
AWS Network Firewall flow logs need direct delivery to Firehose. Which logging configuration selects it?
Set LogDestinationType to KinesisDataFirehose and provide the Firehose stream name under the deliveryStream key.
Can a Firehose stream with dynamic partitioning use zero buffering?
No. Zero buffering is unavailable with dynamic partitioning and for the S3 backup destination.
A Firehose Lambda transformation returns new recordId values for transformed records. What happens?
Firehose treats the mismatch as transformation failure. Preserve each original recordId and return result and base64-encoded data.
What do Ok, Dropped, and ProcessingFailed mean in a Firehose transformation response?
Ok means transformed successfully, Dropped means intentionally discarded, and ProcessingFailed means transformation failed. Ok and Dropped count as successful processing.