Amazon EMR provides a managed Apache Hadoop framework that makes it straightforward, fast, and cost-effective to run Apache HBase. Apache HBase is a massively scalable, distributed big data store in the Apache Hadoop ecosystem. It is an open-source, non-relational, versioned database that runs on top of the Apache Hadoop Distributed File System (HDFS). It’s built for random, strictly consistent, real-time access for tables with billions of rows and millions of columns. Monitoring HBase clusters is critical in order to identify stability and performance bottlenecks and proactively preempt them. In this post, we discuss how you can use Amazon Managed Service for Prometheus and Amazon Managed Grafana to monitor, alert, and visualize HBase metrics.
HBase has built-in support for exporting metrics via the Hadoop metrics subsystem to files or Ganglia or via JMX. You can either use AWS Distro for OpenTelemetry or Prometheus JMX exporters to collect metrics exposed by HBase. In this post, we show how to use Prometheus exporters. These exporters behave like small webservers that convert internal application metrics to Prometheus format and serve it at /metrics path. A Prometheus server running on an Amazon Elastic Compute Cloud (Amazon EC2) instance collects these metrics and remote writes to an Amazon Managed Service for Prometheus workspace. We then use Amazon Managed Grafana to create dashboards and view these metrics using an Amazon Managed Service for Prometheus workspace as its data source.
This solution can be extended to other big data platforms such as Apache Spark and Apache Presto that also use JMX to expose their metrics.
Solution overview
The following diagram illustrates our solution architecture.
This post uses an AWS CloudFormation template to perform below actions:
- Install an open-source Prometheus server on an EC2 instance.
- Create appropriate AWS Identity and Access Management (IAM) roles and security group for the EC2 instance running the Prometheus server.
- Create an EMR cluster with an HBase on Amazon S3 configuration.
- Install JMX exporters on all EMR nodes.
- Create additional security groups for the EMR master and worker nodes to connect with the Prometheus server running on the EC2 instance.
- Create a workspace in Amazon Managed Service for Prometheus.
Prerequisites
To implement this solution, make sure you have the following prerequisites:
- An AWS account that provides access to AWS services.
- AWS IAM Identity Center (successor to AWS Single Sign-On) enabled in your account and an IAM Identity Center user to use with Amazon Managed Grafana. For instructions, refer to Enable IAM Identity Center.
- A key pair to SSH into the EMR master node. For instructions, refer to Create a key pair using Amazon EC2.
- Amazon EMR default roles (
EMR\_DefaultRoleandEMR\_EC2\_DefaultRole). Refer to Configure IAM service roles for Amazon EMR permissions to AWS services and resources for instructions or run the following API from the terminal or AWS Cloud9 to create the default roles:
Deploy the CloudFormation template
Deploy the CloudFormation template in the us-east-1 Region:
It will take 15–20 minutes for the template to complete. The template requires the following fields:
- Stack Name – Enter a name for the stack
- VPC – Choose an existing VPC
- Subnet – Choose an existing subnet
- EMRClusterName – Use
EMRHBase - HBaseRootDir – Provide a new HBase root directory (for example,
s3://hbase-root-dir/). - MasterInstanceType – Use m5x.large
- CoreInstanceType – Use m5x.large
- CoreInstanceCount – Enter 2
- SSHIPRange – Use
<your ip address>/32(you can go to https://checkip.amazonaws.com/ to check your IP address) - EMRKeyName – Choose a key pair for the EMR cluster
- EMRRleaseLabel – Use
emr-6.9.0 - InstanceType – Use the EC2 instance type for installing the Prometheus server
Enable remote writes on the Prometheus server
The Prometheus server is running on an EC2 instance. You can find the instance hostname in the CloudFormation stack’s Outputs tab for key PrometheusServerPublicDNSName.
- SSH into the EC2 instance using the key pair:
- Copy the value for Endpoint – remote write URL from the Amazon Managed Service for Prometheus workspace console.
- Edit
remote\_write urlin/etc/prometheus/conf/prometheus.yml:
It should look like the following code:
- Now we need to restart the Prometheus server to pick up the changes:
Enable Amazon Managed Grafana to read from an Amazon Managed Service for Prometheus workspace
We need to add the Amazon Managed Prometheus workspace as a data source in Amazon Managed Grafana. You can skip directly to step 3 if you already have an existing Amazon Managed Grafana workspace and want to use it for HBase metrics.
- First, let’s create a workspace on Amazon Managed Grafana. You can follow the appendix to create a workspace using the Amazon Managed Grafana console or run the following API from your terminal (provide your role ARN):
- On the Amazon Managed Grafana console, choose Configure users and select a user you want to allow to log in to Grafana dashboards.
Make sure your IAM Identity Center user type is admin. We need this to create dashboards. You can assign the viewer role to all the other users.
- Log in to the Amazon Managed Grafana workspace URL using your admin credentials.
- Choose AWS Data Sources in the navigation pane.
- For Service, choose Amazon Managed Service for Prometheus.
- For Regions, choose US East (N. Virginia).
Create an HBase dashboard
Grafana labs has an open-source dashboard that you can use. For example, you can follow the guidance from the following HBase dashboard. Start creating your dashboard and chose the import option. Provide the URL of the dashboard or enter 12722 and choose Load. Make sure your Prometheus workspace is selected on the next page. You should see HBase metrics showing up on the dashboard.
Key HBase metrics to monitor
HBase has a wide range of metrics for HMaster and RegionServer. The following are a few important metrics to keep in mind.
| HMASTER | Metric Name | Metric Description |
| . | hadoop\_HBase\_numregionservers | Number of live region servers |
| . | hadoop\_HBase\_numdeadregionservers | Number of dead region servers |
| . | hadoop\_HBase\_ritcount | Number of regions in transition |
| . | hadoop\_HBase\_ritcountoverthreshold | Number of regions that have been in transition longer than a threshold time (default: 60 seconds) |
| . | hadoop\_HBase\_ritduration\_99th\_percentile | Maximum time taken by 99% of the regions to remain in transition state |
| REGIONSERVER | Metric Name | Metric Description |
| . | hadoop\_HBase\_regioncount | Number of regions hosted by the region server |
| . | hadoop\_HBase\_storefilecount | Number of store files currently managed by the region server |
| . | hadoop\_HBase\_storefilesize | Aggregate size of the store files |
| . | hadoop\_HBase\_hlogfilecount | Number of write-ahead logs not yet archived |
| . | hadoop\_HBase\_hlogfilesize | Size of all write-ahead log files |
| . | hadoop\_HBase\_totalrequestcount | Total number of requests received |
| . | hadoop\_HBase\_readrequestcount | Number of read requests received |
| . | hadoop\_HBase\_writerequestcount | Number of write requests received |
| . | hadoop\_HBase\_numopenconnections | Number of open connections at the RPC layer |
| . | hadoop\_HBase\_numactivehandler | Number of RPC handlers actively servicing requests |
| Memstore | . | . |
| . | hadoop\_HBase\_memstoresize | Total memstore memory size of the region server |
| . | hadoop\_HBase\_flushqueuelength | Current depth of the memstore flush queue (if increasing, we are falling behind with clearing memstores out to Amazon S3) |
| . | hadoop\_HBase\_flushtime\_99th\_percentile | 99th percentile latency for flush operation |
| . | hadoop\_HBase\_updatesblockedtime | Number of milliseconds updates have been blocked so the memstore can be flushed |
| Block Cache | . | . |
| . | hadoop\_HBase\_blockcachesize | Block cache size |
| . | hadoop\_HBase\_blockcachefreesize | Block cache free size |
| . | hadoop\_HBase\_blockcachehitcount | Number of block cache hits |
| . | hadoop\_HBase\_blockcachemisscount | Number of block cache misses |
| . | hadoop\_HBase\_blockcacheexpresshitpercent | Percentage of the time that requests with the cache turned on hit the cache |
| . | hadoop\_HBase\_blockcachecounthitpercent | Percentage of block cache hits |
| . | hadoop\_HBase\_blockcacheevictioncount | Number of block cache evictions in the region server |
| . | hadoop\_HBase\_l2cachehitratio | Local disk-based bucket cache hit ratio |
| . | hadoop\_HBase\_l2cachemissratio | Bucket cache miss ratio |
| Compaction | . | . |
| . | hadoop\_HBase\_majorcompactiontime\_99th\_percentile | Time in milliseconds taken for major compaction |
| . | hadoop\_HBase\_compactiontime\_99th\_percentile | Time in milliseconds taken for minor compaction |
| . | hadoop\_HBase\_compactionqueuelength | Current depth of the compaction request queue (if increasing, we are falling behind with storefile compaction) |
| . | flush queue length | Number of flush operations waiting to be processed in the region server (a higher number indicates flush operations are slow) |
| IPC Queues | . | . |
| . | hadoop\_HBase\_queuesize | Total data size of all RPC calls in the RPC queues in the region server |
| . | hadoop\_HBase\_numcallsingeneralqueue | Number of RPC calls in the general processing queue in the region server |
| . | hadoop\_HBase\_processcalltime\_99th\_percentile | 99th percentile latency for RPC calls to be processed in the region server |
| . | hadoop\_HBase\_queuecalltime\_99th\_percentile | 99th percentile latency for RPC calls to stay in the RPC queue in the region server |
| JVM and GC | . | . |
| . | hadoop\_HBase\_memheapusedm | Heap used |
| . | hadoop\_HBase\_memheapmaxm | Total heap |
| . | hadoop\_HBase\_pausetimewithgc\_99th\_percentile | Pause time in milliseconds |
| . | hadoop\_HBase\_gccount | Garbage collection count |
| . | hadoop\_HBase\_gctimemillis | Time spent in garbage collection, in milliseconds |
| Latencies | . | . |
| . | HBase.regionserver.<op>\_<measure> | Operation latencies, where <op> is Append, Delete, Mutate, Get, Replay, or Increment, and <measure> is min, max, mean, median, 75th\_percentile, 95th\_percentile, or 99th\_percentile |
| . | HBase.regionserver.slow<op>Count | Number of operations we thought were slow, where <op> is one of the preceding list |
| Bulk Load | . | . |
| . | Bulkload\_99th\_percentile | hadoop\_HBase\_bulkload\_99th\_percentile |
| I/O | . | . |
| . | FsWriteTime\_99th\_percentile | hadoop\_HBase\_fswritetime\_99th\_percentile |
| . | FsReadTime\_99th\_percentile | hadoop\_HBase\_fsreadtime\_99th\_percentile |
| Exceptions | . | . |
| . | exceptions.RegionTooBusyException | . |
| . | exceptions.callQueueTooBig | . |
| . | exceptions.NotServingRegionException | . |
Considerations and limitations
Note the following when using this solution:
- You can set up alerts on Amazon Managed Service for Prometheus and visualize them in Amazon Managed Grafana.
- This architecture can be easily extended to include other open-source frameworks such as Apache Spark, Apache Presto, and Apache Hive.
- Refer to the pricing details for Amazon Managed Service for Prometheus and Amazon Managed Grafana.
- These scripts are for guidance purposes only and aren’t ready for production deployments. Make sure to perform thorough testing.
Clean up
To avoid ongoing charges, delete the CloudFormation stack and workspaces created in Amazon Managed Grafana and Amazon Managed Service for Prometheus.
Conclusion
In this post, you learned how to monitor EMR HBase clusters and set up dashboards to visualize key metrics. This solution can serve as a unified monitoring platform for multiple EMR clusters and other applications. For more information on EMR HBase, see Release Guide and HBase Migration whitepaper.
Appendix
Complete the following steps to create a workspace on Amazon Managed Grafana:
- Log in to the Amazon Managed Grafana console and choose Create workspace.
- For Authentication access, select AWS IAM Identity Center.
If you don’t have IAM Identity Center enabled, refer to Enable IAM Identity Center.
- Optionally, to view Prometheus alerts in your Grafana workspace, select Turn Grafana alerting on.
- On the next page, select Amazon Managed Service for Prometheus as the data source.
- After the workspace is created, assign users to access Amazon Managed Grafana.
- For a first-time setup, assign admin privileges to the user.
You can add other users with only viewer access.
Make sure you are able to log in to the Grafana workspace URL using your IAM Identity Center user credentials.
About the Author
Anubhav Awasthi is a Sr. Big Data Specialist Solutions Architect at AWS. He works with customers to provide architectural guidance for running analytics solutions on Amazon EMR, Amazon Athena, AWS Glue, and AWS Lake Formation.