Big Data Processing with Amazon EMR - Execution Environments for Spark and Hive
This article introduces running Spark jobs and Hive queries on EMR clusters, choosing between them and EMR Serverless, and cost optimization through managed scaling.
EMR Cluster Configuration
An EMR cluster consists of a master node (cluster management and the YARN resource manager), core nodes (HDFS data storage and compute processing), and task nodes (compute processing only). Because core nodes hold HDFS data, there is a risk of data loss when scaling them in, but task nodes hold no data and can therefore scale freely. When you use S3 as primary storage, you limit core node HDFS to temporary storage of intermediate data and reduce costs by using Spot Instances for task nodes. With EC2 instance fleets, you can specify multiple instance types to increase Spot availability.
Running Spark and Hive
With Spark on EMR, you submit jobs with the spark-submit command or the EMR Steps API. EMRFS is the file system EMR uses to read from and write to S3, and because S3 changed from eventual consistency to strong consistency from December 2020 onward, the EMRFS consistent view is no longer needed. When you enable Spark's Dynamic Resource Allocation, the number of executors is adjusted automatically according to the job's load. With Hive on EMR, you can configure the Glue Data Catalog as an external metastore and share table definitions with Athena and Redshift Spectrum. EMR Serverless requires no cluster provisioning, and you run jobs by specifying resources per application.
EMR on EKS and Managed Scaling
EMR on EKS runs Spark jobs on an existing EKS cluster and takes advantage of Kubernetes resource management and scheduling. You create a virtual cluster, map it to an EKS namespace, and submit jobs with the StartJobRun API. Managed scaling for EMR on EC2 automatically adds / removes core nodes and task nodes according to the job load. You set the minimum / maximum number of nodes for scaling, and scaling decisions are made based on YARN memory utilization. EMR Studio is a browser-based IDE that lets you connect to EMR clusters from Jupyter notebooks and run interactive analysis.
EMR Cost Optimization
EMR costs consist of instance charges and EMR charges. You can use a configuration in which task nodes use Spot Instances and core nodes run On-Demand to protect HDFS data. In an EMRFS architecture that uses S3 as primary storage, you can keep core node HDFS to a minimum and reduce the risk of Spot interruptions. A design that launches transient clusters only when jobs run and terminates them automatically on completion keeps the cost of idle time down. Graviton instances (m6g, r6g) can be used to run Spark jobs.
Storage and Data Lake Integration
The basis of using EMR efficiently is separating compute from storage. If you keep data in S3 and launch a cluster to read it only when processing is needed, you do not have to keep the cluster running at all times and can keep costs down. Because the data remains in S3 even after the cluster is terminated, multiple processes can reuse the same data. If you register table definitions in a data catalog, not only EMR but also other analytics services can reference the same data, and you can handle the entire data lake consistently. A design that consolidates storage in persistent S3 is one way to build a big data platform.
Cluster Execution Modes
You use EMR clusters in different ways depending on the purpose. Transient use, in which a cluster runs a defined process and is discarded when it finishes, is suited to batch processing and lets you avoid wasteful running time. For interactive analysis and continuous processing, you use a long-running cluster. You can also define processing as steps and have them run in order. If you choose a serverless execution mode, cluster management itself becomes unnecessary, and you can focus only on submitting jobs. With a notebook environment, data scientists can carry out analysis interactively. You choose the execution mode according to the nature and frequency of the processing.
Optimizing Performance and Cost
EMR cost and performance change depending on how you choose instances. If you use a configuration that flexibly combines multiple instance types, you can make use of low-cost Spot Instances while maintaining availability. Using Spot for processing that can tolerate interruptions lets you keep costs down. With a mechanism that automatically adjusts the cluster size according to load, you can increase and decrease resources to match waves in processing volume and keep waste down. If you plan appropriately for how data is partitioned and how many resources the processing needs, it becomes easier to keep processing time and costs down.
Security and Governance
On a big data platform, protecting the data you handle and managing access to it are important. You encrypt stored data and in-transit communication during processing to protect sensitive information. You combine data access with a fine-grained permission management mechanism and control the scope each user can view. If you restrict access at the column or row level, you can change the scope shown to each department even for the same dataset. You record who processed which data so that it can be audited. Precisely because it is an environment where many users handle large amounts of data, putting in place governance that combines encryption / access control / auditing is the prerequisite for a secure analytics platform.
Summary
EMR provides a managed execution environment for big data frameworks such as Spark and Hive. With an EMRFS architecture that uses S3 as primary storage, you reduce the risk of Spot Instance interruptions, and managed scaling automatically adjusts the number of nodes according to job load. With EMR on EKS, integration with existing Kubernetes environments is also possible.
References (Official AWS Resources)
The primary sources for this page are the official AWS website and documentation. Check the official pages below for the latest specifications and pricing.
If this page and the official documentation disagree, treat the official documentation as authoritative.