Analyze Amazon EMR on Amazon EC2 cluster utilization with Amazon Athena and Amazon QuickSight

Gaining granular visibility into application-level prices on Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2) clusters presents a possibility for purchasers searching for methods to additional optimize useful resource utilization and implement truthful price allocation and chargeback fashions. By breaking down the utilization of particular person functions working in your EMR cluster, you may unlock a number of advantages:

• Knowledgeable workload administration – Software-level price insights empower organizations to prioritize and schedule workloads successfully. Useful resource allocation selections could be made with a greater understanding of price implications, probably enhancing general cluster efficiency and cost-efficiency.
• Price optimization – With granular price attribution, organizations can establish cost-saving alternatives for particular person functions. They’ll right-size underutilized sources or prioritize optimization efforts for functions which might be driving excessive utilization and prices.
• Clear billing – In multi-tenant environments, organizations can implement truthful and clear price allocation fashions primarily based on particular person software useful resource consumption and related prices. This fosters accountability and allows correct chargebacks to tenants.

On this put up, we information you thru deploying a complete answer in your Amazon Net Providers (AWS) atmosphere to research Amazon EMR on EC2 cluster utilization. By utilizing this answer, you’ll acquire a deep understanding of useful resource consumption and related prices of particular person functions working in your EMR cluster. This can provide help to optimize prices, implement truthful billing practices, and make knowledgeable selections about workload administration, finally enhancing the general effectivity and cost-effectiveness of your Amazon EMR atmosphere. This answer has been solely examined on Spark workloads working on EMR on EC2 that makes use of YARN as its useful resource supervisor. It hasn’t been examined on workloads from different frameworks that run on YARN, reminiscent of HIVE or TEZ.

Resolution overview

The answer works by working a Python script on the EMR cluster’s major node to gather metrics from the YARN useful resource supervisor and correlate them with price utilization particulars from the AWS Price and Utilization Experiences (AWS CUR). The script activated by a cronjob makes HTTP requests to the YARN useful resource supervisor to gather two sorts of metrics from paths /ws/v1/cluster/metrics for cluster metrics and /ws/v1/cluster/apps for software metrics. The cluster metrics comprise utilization info of cluster sources, and the appliance metrics comprise utilization info of an software or job. These metrics are saved in an Amazon Easy Storage Service (Amazon S3) bucket.

There are two YARN metrics that seize the useful resource utilization info of an software or job.

• memorySeconds – That is the reminiscence (in MB) allotted to an software occasions the variety of seconds the appliance ran
• vcoreSeconds – That is the variety of YARN vcores allotted to an software occasions the variety of seconds software ran

The answer makes use of memorySeconds to derive the price of working the appliance or job. It may be modified to make use of vcoreSeconds as a substitute if essential.

The metadata of the YARN metrics collected in Amazon S3 is created, saved, and represented as database and tables in AWS Glue Knowledge Catalog, which is in flip out there to Amazon Athena for additional processing. Now you can write SQL queries in Athena to correlate the YARN metrics with the associated fee utilization info from AWS CUR to derive the detailed price breakdown of your EMR cluster by infrastructure and software. This answer creates two corresponding Athena views of the respective price breakdown that can develop into the information supply to Amazon QuickSight for visualization.

The next diagram reveals the answer structure.

Conditions

To carry out the answer, you want the next conditions:

1. Verify {that a} CUR is created in your AWS account. It wants an S3 bucket to retailer the report information. Comply with the steps described in Creating Price and Utilization Experiences to create the CUR on the AWS Administration Console. When creating the report, ensure the next settings are enabled:

• • Embody useful resource IDs
  • Time granularity is ready to hourly
  • Report information integration to Athena

It could possibly take as much as 24 hours for AWS to start out delivering experiences to your S3 bucket. Thereafter, your CUR will get up to date at the very least one time a day.

2. The answer wants Athena to run queries in opposition to the information from the CUR utilizing commonplace SQL. To automate and streamline the combination of Athena with CUR, AWS supplies an AWS CloudFormation template, crawler-cfn.yml, which is mechanically generated in the identical S3 bucket throughout CUR creation. Comply with the directions in Establishing Athena utilizing AWS CloudFormation templates to combine Athena with the CUR. This template will create an AWS Glue database that references to the CUR, an AWS Lambda occasion and an AWS Glue crawler that will get invoked by S3 occasion notification to replace the AWS Glue database at any time when the CUR will get up to date.
3. Be sure that to activate the AWS generated price allocation tag, aws:elasticmapreduce:job-flow-id. This allows the sphere, resource_tags_aws_elasticmapreduce_job_flow_id, within the CUR to be populated with the EMR cluster ID and is utilized by the SQL queries within the answer. To activate the associated fee allocation tag from the administration console, comply with these steps:
   • Register to the payer account’s AWS Administration Console and open the AWS Billing and Price Administration console
   • Within the navigation pane, select Price Allocation Tags
   • Underneath AWS generated price allocation tags, select the aws:elasticmapreduce:job-flow-id tag
   • Select Activate. It could possibly take as much as 24 hours for tags to activate.

The next screenshot reveals an instance of the aws:elasticmapreduce:job-flow-id tag being activated.

Now you can check out this answer on an EMR cluster in a lab atmosphere. When you’re not already accustomed to EMR, comply with the detailed directions supplied in Tutorial: Getting began with Amazon EMR to launch a brand new EMR cluster and run a pattern Spark job.

Deploying the answer

To deploy the answer, comply with the steps within the subsequent sections.

Putting in scripts to the EMR cluster

Obtain two scripts from the GitHub repository and save them into an S3 bucket:

• emr_usage_report.py – Python script that makes the HTTP requests to YARN Useful resource Supervisor
• emr_install_report.sh  – Bash script that creates a cronjob to run the python script each minute

To put in the scripts, add a step to the EMR cluster by the console or AWS Command Line Interface (AWS CLI) utilizing aws emr add-step command.

Change:

• REGION with the AWS Areas the place the cluster is working (for instance, Europe (Eire) eu-west-1)
• MY-BUCKET with the identify of the bucket the place the script is saved (for instance, my.artifact.bucket)
• MY_REPORT_BUCKET with the bucket identify the place you wish to accumulate YARN metrics (for instance, my.report.bucket)

aws emr add-steps 
--cluster-id j-XXXXXXXXXXXXX 
--steps Kind=CUSTOM_JAR,Identify="Set up YARN reporter",Jar=s3://REGION.elasticmapreduce/libs/script-runner/script-runner.jar,Args=[s3://<MY-BUCKET>/emr-install_reporter.sh,s3://<MY-BUCKET>/emr_usage_reporter.py,MY_REPORT_BUCKET]

Now you can run some Spark jobs in your EMR cluster to start out producing software utilization metrics.

Launching the CloudFormation stack

When the conditions are met and you’ve got the scripts deployed in order that your EMR clusters are sending YARN metrics to an S3 bucket, the remainder of the answer could be deployed utilizing CloudFormation.

Earlier than launching the stack, add a duplicate of this QuickSight definition file into an S3 bucket required by the CloudFormation template to construct the preliminary evaluation in QuickSight. When prepared, proceed to launch your stack to provision the remaining sources of the answer.

1. Select
   

This mechanically launches AWS CloudFormation in your AWS account with a template. It prompts you to check in as wanted and be sure you create the stack in your meant Area.

The CloudFormation stack requires just a few parameters, as proven within the following screenshot.

The next desk describes the parameters.

2. Enter the parameter values from the previous desk.
3. Select Subsequent.
4. On the subsequent display, enter any essential tags, an AWS Identification and Entry Administration (IAM) position, stack failure, or superior choices if essential. In any other case, you may go away them as default.
5. Select Subsequent.
6. Evaluation the main points on the ultimate display and choose the test packing containers confirming AWS CloudFormation may create IAM sources with customized names or require CAPABILITY_AUTO_EXPAND.
   
7. Select Create.

The stack will take a few minutes to create the remaining sources for the answer. After the CloudFormation stack is created, on the Outputs tab, yow will discover the main points of the sources created.

Reviewing the correlation outcomes

The CloudFormation template creates two Athena views containing the correlated price breakdown particulars of the YARN cluster and software metrics with the CUR. The CUR aggregates price hourly and due to this fact correlation to derive the price of working an software is prorated primarily based on the hourly working price of the EMR cluster.

The next screenshot reveals the Athena view for the correlated price breakdown particulars of YARN cluster metrics.

The next desk describes the fields within the Athena view for YARN cluster metrics.

The next screenshot reveals the Athena view for the correlated price breakdown particulars of YARN software metrics.

The next desk describes the fields within the Athena view for YARN software metrics.

Constructing your personal visualization

In QuickSight, the CloudFormation template creates two datasets that reference Athena views as information sources and a pattern evaluation. The pattern evaluation has two sheets, EMR Infra Spend and EMR App Spend. They’ve a prepopulated bar chart and pivot tables to exhibit how you should utilize the datasets to construct your personal visualization to current the associated fee breakdown particulars of your EMR clusters.

EMR Infra Spend sheet references to the YARN cluster metrics dataset. There’s a filter for date vary choice and a filter for cluster ID choice. The pattern bar chart reveals the consolidated price breakdown of the sources for every cluster in the course of the interval. The pivot desk breaks them down additional to point out their day by day expenditure.

The next screenshot reveals the EMR Infra Spend sheet from pattern evaluation created by the CloudFormation template.

EMR App Spend sheet references to the YARN software metrics. There’s a filter for date vary choice and a filter for cluster ID choice. The pivot desk on this sheet reveals how you should utilize the fields within the dataset to current the associated fee breakdown particulars of the cluster by customers to look at the functions that had been run, whether or not they had been accomplished efficiently or not, the time and period of every run, and the derived price of the run.

The next screenshot reveals the EMR App Spend sheet from pattern evaluation created by the CloudFormation template.

Cleanup

When you now not want the sources you created throughout this walkthrough, delete them to stop incurring extra fees. To wash up your sources, full the next steps:

1. On the CloudFormation console, delete the stack that you simply created utilizing the template
2. Terminate the EMR cluster
3. Empty or delete the S3 bucket used for YARN metrics

Conclusion

On this put up, we mentioned tips on how to implement a complete cluster utilization reporting answer that gives granular visibility into the useful resource consumption and related prices of particular person functions working in your Amazon EMR on EC2 cluster. By utilizing the facility of Athena and QuickSight to correlate YARN metrics with price utilization particulars out of your Price and Utilization Report, this answer empowers organizations to make knowledgeable selections. With these insights, you may optimize useful resource allocation, implement truthful and clear billing fashions primarily based on precise software utilization, and finally obtain higher cost-efficiency in your EMR environments. This answer will provide help to unlock the total potential of your EMR cluster, driving steady enchancment in your information processing and analytics workflows whereas maximizing return on funding.

Concerning the authors

Boon Lee Eu is a Senior Technical Account Supervisor at Amazon Net Providers (AWS). He works intently and proactively with Enterprise Help prospects to supply advocacy and strategic technical steerage to assist plan and obtain operational excellence in AWS atmosphere primarily based on greatest practices. Primarily based in Singapore, Boon Lee has over 20 years of expertise in IT & Telecom industries.

Kyara Labrador is a Sr. Analytics Specialist Options Architect at Amazon Net Providers (AWS) Philippines, specializing in huge information and analytics. She helps prospects in designing and implementing scalable, safe, and cost-effective information options, in addition to migrating and modernizing their huge information and analytics workloads to AWS. She is keen about empowering organizations to unlock the total potential of their information.

Vikas Omer is the Head of Knowledge & AI Resolution Structure for ASEAN at Amazon Net Providers (AWS). With over 15 years of expertise within the information and AI house, he’s a seasoned chief who leverages his experience to drive innovation and enlargement within the area. Vikas is keen about serving to prospects and companions succeed of their digital transformation journeys, specializing in cloud-based options and rising applied sciences.

Lorenzo Ripani is a Large Knowledge Resolution Architect at AWS. He’s keen about distributed techniques, open supply applied sciences and safety. He spends most of his time working with prospects around the globe to design, consider and optimize scalable and safe information pipelines with Amazon EMR.