AWS Data Pipeline schedules data-driven workflows that copy and transform data on EC2 and EMR. It is in maintenance mode and closed to new customers.
Legacy data workflows
AWS Data Pipeline still appears in older exam questions about scheduled copies and EMR jobs. Today it only serves existing customers, so new designs use AWS Glue, AWS Step Functions or Amazon MWAA.
How it works
- A pipeline definition describes activities, data nodes, schedules and preconditions.
- The pipeline launches EC2 instances (or EMR clusters) on schedule.
- Task Runner on those instances polls for tasks and runs them in dependency order.
| Workload | Replacement |
|---|---|
| Spark or Python ETL, crawlers, templates such as DynamoDB to S3 | AWS Glue |
| Orchestrating EMR, Lambda, Glue, DynamoDB steps | AWS Step Functions |
| Existing Apache Airflow skills or DAGs | Amazon MWAA |
Exam cues
- If the question names Data Pipeline for a new solution, think AWS Glue or Step Functions, because Data Pipeline is closed to new customers.
- If the question is older and asks for scheduled data movement with EMR, Data Pipeline may be the expected answer.
Remember
- Maintenance mode; no new customers (guide entry July 25, 2025).
- Replacements: Glue, Step Functions, MWAA.
Trap: “Scheduled ETL” doesn’t mean Data Pipeline anymore. For a new account, AWS Glue jobs and triggers or Step Functions do the job.
Da documentação da AWS (6)
- AWS Data Pipeline automates the movement and transformation of data with data-driven workflows in which tasks can depend on earlier tasks succeeding.
- A Data Pipeline pipeline schedules and runs tasks by creating Amazon EC2 instances, where Task Runner polls for and performs the work, such as copying logs to S3 or launching EMR clusters.
- AWS Data Pipeline is in maintenance mode and no longer available to new customers (recorded in its guide on July 25, 2025); existing customers continue as normal.
- AWS recommends migrating Data Pipeline workloads to AWS Glue, AWS Step Functions or Amazon MWAA, depending on the workload.
- AWS Glue suits migrated Data Pipeline workloads that need serverless data integration, Spark or Python jobs and crawlers, such as exporting a DynamoDB table to Amazon S3.
- AWS Step Functions suits migrated Data Pipeline workloads that orchestrate tasks across services such as EMR, Lambda, Glue and DynamoDB, billed per task execution.
Flashcards
Nada para revisar aqui agora. Escolha outros tópicos ou outro filtro.
·
Todos os 4 flashcards em lista
A new AWS customer asks for AWS Data Pipeline to export a DynamoDB table to S3 every night. What should they use instead?
AWS Glue (or Step Functions for orchestration).
Data Pipeline is closed to new customers and in maintenance mode.
A team is moving an AWS Data Pipeline data integration job that uses Spark and Python. Which service is a suitable replacement?
AWS Glue supports serverless data integration with Spark and Python jobs.
An existing AWS Data Pipeline workflow moves and transforms data through dependent tasks. What does the service automate?
AWS Data Pipeline automates data movement and transformation, with tasks that can depend on earlier tasks succeeding.
A Data Pipeline workflow coordinates EMR steps, Lambda functions and DynamoDB writes. Which service is the recommended serverless replacement for the orchestration?
AWS Step Functions.