Welcome to our AWS Certified Solutions Architect Professional tutorial series, and in this tutorial, we’re going to take a deep dive into AWS EMR (Elastic MapReduce). This comprehensive course will cover everything you need to know to master AWS EMR, from configuring and spinning up EMR clusters to running Spark ETL jobs. We’ll also explore notebooks, delve into Hive and Pig, explore step functions for orchestration, and touch on auto-scaling.
Before we dive into the content, I want to let you know that all the resources you’ll need for this course are available in the description below. This includes links to GitHub for the code and Buy Me a Coffee for the slides, and guess what? It’s all completely free!
Now, let’s get started with the course. But before we jump into the technical details, we have some housekeeping to take care of. So, let’s head over to the computer and begin the setup.
Setup: Alright, folks, we’re all set to start the setup process. I’ve logged into the AWS console, and the first thing we need to do is create a new Virtual Private Cloud (VPC). Let’s go ahead and do that:
- Type “VPC” in the search bar at the top, and select “VPC” under “Isolated Cloud Resources.”
- Click on “Launch VPC Wizard.”
- We’ll leave most settings as default, but let’s give our VPC a name. I’ll call it “emr-tutorial.”
- For the number of public subnets, you can keep it as the default (2). We only need one, but that’s fine.
- Set the number of private subnets to 0 since we won’t need any for this tutorial.
- Make sure there’s a NAT Gateway with the S3 option selected.
- Optionally, you can change the name to something more meaningful.
- Click “Create VPC.”
The VPC setup can be a bit confusing with the new wizard, but we’ve got our VPC ready now. Next, we need to set up a Cloud9 development environment. Cloud9 provides free IDEs (Integrated Development Environments) by AWS, and I’ll be using a free EC2 instance type, so it won’t cost us anything.
- Type “Cloud9” in the search bar at the top and select “Cloud9.”
- Click “Create environment.”
- Give your environment a name; I’ll call it “emr-tutorial” as well.
- Choose the default settings, making sure it’s set to automatically stop after 30 minutes to avoid costs.
- Under “Network settings,” select the VPC you just created (in my case, “emr-tutorial VPC”).
- Keep the subnet as “1b” if that’s what you selected for your Cloud9 instance.
- Click “Next step.”
- Choose “Create a new EC2 instance for environment (direct access).”
- Make sure the instance type is “t2.micro” to stay within the free tier.
- Leave other settings as default and click “Next step.”
- Review your settings and click “Create environment.”
The Cloud9 environment will take a moment to set up. Once it’s ready, you can access it to continue with the setup.
Now, let’s move on to creating an SSH key pair for secure access:
- Go to EC2 by typing “EC2” in the search bar and selecting “EC2.”
- In the left menu, navigate to “Key Pairs” under “Network & Security.”
- Click “Create Key Pair.”
- Give your key pair a name, like “emr-tutorial.”
- Choose the “RSA” key pair type and download the private key file (e.g., “emr-tutorial.pem”).
- Keep this file secure, as it will be used for SSH access to your instances.
With the SSH key pair in hand, we’ll upload it to your Cloud9 environment:
- Return to your Cloud9 environment.
- In the Cloud9 environment, click “File” and then select “Upload Local Files.”
- Select the private key file you just downloaded (e.g., “emr-tutorial.pem”).
- The key file will appear in your Cloud9 environment.
Now, let’s make sure no one else can access the key file by running the following command:
bashCopy codechmod 400 emr-tutorial.pem
This command restricts access to the key file. Now you’re all set up and secure for the AWS EMR tutorial. In the next section, we’ll dive into the core concepts of AWS EMR. Let’s jump back to the console and get started.
Terminology:
Before we start configuring and running EMR clusters, let’s get familiar with some essential AWS EMR terminology:
- EMR (Elastic MapReduce): EMR is a managed clustered platform that simplifies running big data frameworks. It stands for Elastic MapReduce.
- Master Node: The master node manages the cluster by coordinating the distribution of data and tasks among other nodes.
- Core Nodes: Core nodes are responsible for running tasks and storing data in the Hadoop Distributed File System (HDFS) on the cluster. At least one master node and one core node are required for a minimum EMR cluster setup.
- Task Nodes: Task nodes run tasks on data but do not store data themselves. They are optional in an EMR cluster.
- Data Processing Frameworks: EMR supports various data processing frameworks, including Spark, Hive, Pig, and more. These frameworks help process and analyze data on the cluster.
- YARN (Yet Another Resource Negotiator): YARN is a cluster resource management tool that manages resources and job scheduling in EMR clusters.
- Storage: EMR uses HDFS to store data, but many users prefer using the EMR File System (EMRFS) to store data on Amazon S3 for cost-effective and scalable storage.
- Local File System: Each node in the EMR cluster has a local file system, but it’s typically used for the operating system and software, not for data storage.
With these foundational terms in mind, let’s proceed to set up our first EMR cluster. Back to the console!
Setting up EMR Cluster:
In this section, we’ll walk through the steps to set up an EMR cluster for our tutorial. Follow along with these steps:
- Type “EMR” in the search bar at the top and select “EMR” to go to the EMR service.
- Click “Create cluster.”
- Choose “Go to advanced options.”
- Under “Software configuration,” select the following frameworks:
- Hadoop
- JupyterHub
- Hive
- Jupyter Enterprise Gateway
- Hue
- Spark
- Pig
- Livy
- Click “Next.”
- For “Network,” choose the VPC you created earlier.
- For “Subnet,” select a subnet in the same availability zone as your Cloud9 instance (e.g., “us-east-1b”).
- Set the number of master nodes to 1