Easy ReviewDOP-C02
Monitoring and Logging8

OpenSearch

Search and visualize operational logs with managed OpenSearch.

Stream logs into a domain and use OpenSearch Dashboards to explore them.

Log ingestion, search, and dashboards

Amazon OpenSearch Service manages search and analytics clusters for workloads such as application log analysis. OpenSearch Dashboards visualizes the data.

How it works

  1. Create a domainAn OpenSearch Service domain is a managed cluster with its own settings and resources., the managed cluster that stores and searches data. Choose public access or access through a virtual private cloudA VPC is a logically isolated virtual network in which AWS resources can run. (VPC), a logically isolated virtual network.
  2. For CloudWatch Logs, create an OpenSearch subscription filterA subscription filter selects log events by pattern and identifies where to send matching events. on a Standard class log group. The filter selects matching events for near-real-time delivery to the domain. The console setup requires a Lambda execution role with domain access; a VPC domain also needs the AWSLambdaVPCAccessExecutionRole policy.
  3. Use OpenSearch Dashboards to visualize the data. An indexAn OpenSearch index is like a database table, with documents as rows and fields as columns. is like a database table, with documents as rows.

Exam cues

  • If the question says search application logs and build dashboards, think OpenSearch Service and OpenSearch Dashboards, because they provide log analytics and visualization.
  • If the question says stream a CloudWatch log group into OpenSearch, think an OpenSearch subscription filter, because it sends matching events to the domain.
  • If the question says query archived files directly in S3 using SQL, think Athena, because it queries S3 data without an OpenSearch domain.
  • If the question says deliver CloudWatch Logs to OpenSearch through Firehose, think the OpenSearch subscription filter, because Firehose does not support that CloudWatch Logs destination path.

Remember

  • Subscriptions require the Standard log class.
  • Public or VPC domain access cannot be changed after domain creation.

Trap: Firehose supports OpenSearch destinations, but not delivery of CloudWatch Logs to those destinations. CloudWatch combines multiple log events in a single Firehose record.

Search CloudWatch logs
12CloudWatch LogsOpenSearchDashboards
  1. A subscription filter sends matching log events to the domain.

  2. OpenSearch Dashboards visualizes data in the domain.

From the AWS docs (13)
  • OpenSearch Service manages OpenSearch clusters for search and analytics, including log analytics and application monitoring.
  • An OpenSearch Service domain is a cluster with specified settings, instance types, instance counts, and storage.
  • OpenSearch Dashboards visualizes data in OpenSearch Service.
  • An OpenSearch index is comparable to a database table, with documents as rows and fields as columns.
  • A CloudWatch Logs subscription filter can send matching log events to an OpenSearch Service domain in near real time.
  • CloudWatch Logs subscriptions support only Standard class log groups.
  • The CloudWatch console's OpenSearch subscription setup requires a Lambda execution role with access to the domain; a VPC domain also requires the AWSLambdaVPCAccessExecutionRole policy.
  • An OpenSearch Service domain can use public or VPC access, but its access type cannot be changed after creation.
  • A VPC is a logically isolated virtual network in which AWS resources can run.
  • Firehose does not support delivering CloudWatch Logs to an OpenSearch Service destination because CloudWatch combines multiple log events in one Firehose record.
  • Athena runs interactive SQL queries directly against data stored in S3.
  • OpenSearch Serverless collections scale compute automatically, suit infrequent or unpredictable workloads, and are always encrypted.
  • An OpenSearch Ingestion pipeline can stream DynamoDB table changes into OpenSearch, with or without an initial full snapshot, to add search over DynamoDB data.

Ingestion delivery and retries

Firehose delivery to an OpenSearch Service domainAn OpenSearch Service domain is a managed cluster with its own settings and resources. and CloudWatch Logs subscriptions have distinct requirements for record format and delivery errors. Knowing these requirements helps explain how records are indexed and what can happen during subscription delivery. Firehose delivery and CloudWatch Logs subscriptions

How it works

  1. Send UTF-8 encoded, single-line JSON objects to Firehose. It buffers the records and sends bulk indexing requests to the OpenSearch Service domain.
  2. For CloudWatch Logs subscriptions, expect at-least-once deliveryAt-least-once delivery permits duplicate events when delivering a subscription.: events can be delivered more than once. A log group can have up to five subscription filtersA subscription filter selects log events by pattern and identifies where to send matching events..
  3. If subscription delivery encounters a retryable error, CloudWatch Logs retries for up to 24 hours. A non-retryable error temporarily disables the filter, and logs are skipped while it is disabled.

Exam cues

  • If the question says an application sends records directly to Firehose for OpenSearch indexing, think UTF-8 encoded single-line JSON objects, because Firehose sends those records in bulk indexing requests.
  • If the question says subscription events appear more than once, think at-least-once delivery, because duplicates can occur.
  • If the question says subscription delivery encounters a retryable error, think retries for up to 24 hours, because CloudWatch Logs handles retryable and non-retryable errors differently.

Remember

  • A CloudWatch Logs log group supports up to five subscription filters.
  • CloudWatch Logs retries retryable subscription delivery errors for up to 24 hours.
  • Firehose records for OpenSearch Service must be UTF-8 encoded single-line JSON objects.

Trap: At-least-once delivery can produce duplicates. A non-retryable subscription error temporarily disables the filter, and logs are skipped during that period.

From the AWS docs (3)
  • Firehose buffers records and sends bulk indexing requests to an OpenSearch Service domain; records must be UTF-8 encoded single-line JSON objects.
  • A CloudWatch log group can have up to five subscription filters; delivery of events is at least once and duplicates can occur.
  • CloudWatch Logs retries retryable subscription delivery errors for up to 24 hours; non-retryable errors disable the filter temporarily and logs are skipped during that period.

Flashcards

tap to flip
Answer

·

All 8 cards as a list

A team needs interactive search and dashboards for CloudWatch application logs. What AWS service and ingestion path fit?

OpenSearch Service and OpenSearch Dashboards. Use a CloudWatch Logs subscription filter for a Standard class log group.

A team wants to send CloudWatch Logs through Firehose directly to an OpenSearch Service destination. Is this supported?

No. Firehose does not support that CloudWatch Logs destination path; use the OpenSearch subscription filter.

A team wants to query archived logs directly in S3 using SQL rather than operate a search cluster. Which AWS service fits?

Amazon Athena. It runs interactive SQL queries directly against data in S3.

Can an existing OpenSearch Service domain switch from public access to VPC access?

No. Choose the access type when creating the domain; it cannot be changed afterward.

A team wants to analyze CloudWatch log events in Amazon OpenSearch Service as they arrive. What can send matching events to the domain?

A CloudWatch Logs subscription filter can send matching events to the OpenSearch Service domain in near real time.

A team sends application records directly to Firehose for indexing in OpenSearch Service. What record format is required?

UTF-8 encoded, single-line JSON objects. Firehose buffers records and sends bulk indexing requests.

How many subscription filters can a CloudWatch Logs log group have, and can delivery contain duplicates?

Up to five filters. Delivery is at least once, so duplicates can occur.

How long does CloudWatch Logs retry a retryable error when delivering a subscription to OpenSearch Service?

Up to 24 hours. Non-retryable errors behave differently and can cause logs to be skipped.

You finished every topic of DOP-C02!

Your badge turned gold. Next step: book the exam on AWS. Once you pass, add yourself to Who passed.

Share

Share that you finished this certification

I finished every topic of AWS Certified DevOps Engineer - Professional (DOP-C02) on Easy Review. Come study too!