Easy ReviewDOP-C02
Monitoring and Logging12

EMR

Run distributed data processing frameworks and plan EMR cluster node roles.

Amazon EMR runs frameworks such as Spark and Hadoop for large-scale data processing.

Distributed processing workloads

Amazon EMR is a managed platform for running distributed data processing frameworks such as Apache Spark and Hadoop. The examples below use EMR on EC2, where a cluster is a group of machines working on the data.

EMR Serverless runs big data frameworks without managing clusters or servers. EMR on EKS runs Spark jobs on an EKS cluster without provisioning EMR clusters.

How it works

  1. Choose processing applications and create an EC2 cluster. Each EC2 instance is a nodeAn EC2 instance in an EMR on EC2 cluster..
  2. Bootstrap actionsA script that customizes EMR nodes before cluster applications are installed., scripts that customize nodes, run before EMR installs the selected applications. They also run on nodes added later.
  3. The primary node coordinates work. Core nodes process tasks and store data in HDFSThe distributed file system in which EMR core nodes store cluster data., the Hadoop Distributed File System. Optional task nodes process tasks without HDFS storage.
  4. Submit stepsA unit of work submitted to an EMR cluster for processing by installed software., units of work for the installed applications. Input and output can be in Amazon S3.
  5. Configure the cluster to wait for further work or terminate after the last step.
Node Role Consequence of loss
Primary Coordinates work and monitors health. Losing the only primary ends the cluster; a three-primary cluster can fail over.
Core Runs tasks and stores HDFS data. Termination risks HDFS data loss.
Task Runs tasks without storing HDFS data. Persistent HDFS data is not stored on the node.

Exam cues

  • If the question says managed Spark processing over large datasets, think EMR, because it runs distributed processing frameworks.
  • If the question says Hadoop processing of newline-separated log files in S3, think EMR, because it can read S3 input and Hadoop accepts newline-separated text by default.
  • If the question says data-critical HDFS with cheaper extra compute, think On-Demand primary and core nodes plus SpotAn EC2 instance using spare capacity that EC2 can interrupt when it needs the capacity back. task nodes, because task nodes do not store HDFS data.
  • If the question says tolerate a primary node failure, think three primary nodes, because EMR can fail over to a standby and replace the failed node.
  • If the question says install custom software before EMR applications, think bootstrap action, because it runs before application installation.

Remember

  • With instance groups, EMR 5.23.0 and later support three primary nodes for high availability. Instance fleetsA node configuration that can mix EC2 instance types and purchasing options. require a supported later release.
  • A cluster can specify up to 16 bootstrap actions.
  • EMR 7.10.0, released in August 2025, made S3AA Hadoop connector for reading and writing data in Amazon S3. the default S3 connector. Earlier releases used EMRFSAn EMR file system that lets Hadoop applications read and write files in Amazon S3. for s3:// and s3n://; EMRFS remains configurable.
  • Spot Instances are interruptible EC2 capacity. A Spot task node can be lost without losing persistent HDFS data on that node, but running work may still be affected.
  • In EMR 6.x, YARNA component that manages resources for applications in an EMR cluster. node labels are disabled by default. YARN manages application resources; enable labels and use the CORE expression to restrict application primary processes to core nodes.
  • Instance fleets can mix instance types and On-Demand and Spot capacity.

Trap: Spot task nodes do not store HDFS data, but do not assume application primary processes always run on core nodes. EMR 6.x disables node labels by default.

EMR on EC2 node roles
12Primary nodeCore nodeTasks and HDFSTask nodeTasks only
  1. The primary coordinates core nodes, which process tasks and store HDFS data.

  2. The primary coordinates optional task nodes, which process tasks without HDFS storage.

From the AWS docs (26)
  • Amazon EMR is a managed cluster platform for data processing frameworks including Apache Hadoop and Apache Spark.
  • EMR processing steps can use input in Amazon S3 and write output to S3.
  • The default Hadoop input format is text files with newline-separated lines.
  • An EMR on EC2 cluster is a collection of EC2 instances called nodes.
  • The primary node coordinates cluster tasks and monitors cluster health.
  • Core nodes run tasks and store data in HDFS.
  • Task nodes run tasks without storing data in HDFS, and are optional.
  • Terminating core nodes risks HDFS data loss.
  • Task nodes do not hold persistent HDFS data, making them suitable for Spot capacity.
  • Termination of a single primary node ends the cluster.
  • Clusters with three primary nodes can fail over and replace a failed primary.
  • High availability for instance groups is supported in EMR 5.23.0 and later; instance fleet support starts in specified later releases.
  • For data-critical workloads AWS suggests On-Demand primary and core nodes with Spot task nodes.
  • EC2 can interrupt Spot Instances when it needs the capacity back.
  • Starting with EMR 6.x, YARN node labels are disabled by default and application primary processes can run on core or task nodes.
  • Enabling YARN node labels and setting the application primary label expression to CORE restricts those processes to core nodes.
  • Bootstrap actions run after instance launch and before application installation and data processing, including on nodes added later.
  • A cluster can have up to 16 bootstrap actions.
  • EMR steps are units of work submitted to a cluster for processing by installed software.
  • A cluster can be configured to terminate automatically after its last step.
  • Instance fleets support a mix of instance types and target capacities for On-Demand and Spot Instances.
  • EMR 7.10.0 and later use S3A by default for S3 file schemes; EMRFS can still be configured.
  • Before EMR 7.10.0, s3:// and s3n:// used EMRFS.
  • EMR 7.10.0 was initially released in August 2025.
  • EMR Serverless runs big data frameworks without configuring, managing and scaling clusters or servers.
  • EMR on EKS submits Spark jobs on EKS without provisioning EMR clusters.

Flashcards

tap to flip
Answer

·

All 12 cards as a list

A team needs a managed platform for processing a large dataset with Apache Spark. Which AWS service fits?

Amazon EMR runs data processing frameworks such as Apache Spark and Hadoop.

A team wants to process data stored in Amazon S3 using a managed Hadoop cluster. Which AWS service fits?

Amazon EMR can run Hadoop processing steps that read input from S3 and write output to S3.

An Amazon EMR on EC2 workload uses HDFS for cluster storage. Which nodes both store that data and run processing tasks?

Core nodes. Terminating them risks losing HDFS data.

A data-critical Amazon EMR cluster needs lower-cost processing capacity while keeping HDFS data off interruptible instances. Which capacity mix fits?

Use On-Demand primary and core nodes with Spot task nodes. Task nodes do not store persistent HDFS data.

An Amazon EMR cluster needs extra processing capacity without adding HDFS storage. Which node type fits?

Task nodes run processing tasks without storing data in HDFS. They are optional.

An Amazon EMR cluster must tolerate the failure of a primary node. Which configuration supports failover?

Use a cluster with three primary nodes. It can fail over and replace a failed primary.

A team wants to add interruptible compute to an Amazon EMR cluster that stores data in HDFS. Why are task nodes a suitable choice?

Task nodes do not hold persistent HDFS data, so they are suitable for Spot capacity.

An Amazon EMR cluster needs custom setup on each node before data processing begins. Which feature fits?

Use bootstrap actions. They run after instance launch and before application installation and processing, including on nodes added later.

A nightly Amazon EMR processing job should release its cluster after the work finishes. What option fits?

Configure the cluster to terminate automatically after its last step.

An Amazon EMR processing job needs durable input and output outside its cluster. Which storage service can its steps use?

Amazon S3. EMR processing steps can read input from S3 and write output there.

A team needs Spark processing without configuring or scaling clusters or servers. Which Amazon EMR option fits?

Amazon EMR Serverless runs big data frameworks without configuring, managing or scaling clusters or servers.

A team already runs Amazon EKS and wants to submit Spark jobs without provisioning Amazon EMR clusters. Which EMR option fits?

Amazon EMR on EKS runs Spark jobs on EKS without provisioning EMR clusters.

You finished every topic of DOP-C02!

Your badge turned gold. Next step: book the exam on AWS. Once you pass, add yourself to Who passed.

Share

Share that you finished this certification

I finished every topic of AWS Certified DevOps Engineer - Professional (DOP-C02) on Easy Review. Come study too!