Monitoring AWS Managed Services using Elastic

Updated: Mar 24, 2021

Over the course of years the number of cloud native services provided by the cloud platforms have been ever increasing. The platforms do come in with their inbuilt monitoring services but from an administrative standpoint of view it is not that helpful as it does not come with a consolidated view for the end users. To overcome this and to provide a common platform for various service logs and its corresponding visualisation many third party providers have come into the play such as Elastic, Datadog, Splunk etc.

In this article we will be seeing the use of one such service provider: Elastic.

Elastic provides out of the box dashboards to visualise AWS managed services such as Loadbalancers, VPC, S3, CloudTrail and Billing. We will be using a sample Web application hosted on AWS cloud to provide how consolidated dashboards can be viewed in Kibana.

Monitoring Dashboards for AWS Services

Once we complete the exercise, below are the set of Dashboards that would be available for the Infrastructure Monitoring team to visualise how AWS services are functioning, identify and troubleshoot problems pertaining to any of the services.

Deployment Architecture

Below is a typical web application deployed on AWS cloud that uses a combination of cloud native services and web application server to serve APIs. We also have a static site hosted on Cloudfront to serve static html/css content.

Metricbeat

Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics (cpu usage, Disk I/O, network bytes in/out ) from the operating system and from services running on the server. Metricbeat takes the metrics and statistics that it collects and ships them to the output that you specify, such as Elasticsearch or Logstash. Metricbeat by default supports various pre-built modules but in this article we will be focusing on the AWS module.

Metricbeat collects two broad categories of metrics i.e. host metrics and managed services metrics. We will be dedicating one EC2 machine to run metricbeat for AWS managed services metrics collection and another metricbeat on EC2 machine to demonstrate metrics collected from the host server.

NOTE: We will be focusing on metricbeat agent from 7.4 and later

AWS Module:

The AWS module for metricbeat currently supports out of the box collection of metrics for the following AWS Services:

EC2
Elastic Loadbalancer
Lambda functions
NAT Gateway
RDS
S3 Storage
SNS
SQS
Transit Gateway
VPN
Billing

Before we dive into installing and configuring metricbeat, let's understand how metricbeat collects, stores and sends metrics to Elastic.

Enable Cloudwatch metrics for each of the AWS Managed services. This will start sending AWS services to CloudWatch, from where Metricbeat will be reading. This documentation gives step by step instructions on how to enable Cloudwatch for a specific AWS service.
Metricbeat will be running on a EC2 machines and configured to collect AWS managed services metrics. This is as simple as enabling the AWS module in metricbeat config file.
Hands-off, Metricbeat periodically queries AWS Cloudwatch to read metrics and sends it to Elasticsearch server, where it is indexed and visualised in Kibana.

Prerequisites

Running a Metricbeat requires the following:

AWS account with credentials. Below section covers the Roles required for Metricbeat to read metrics.
Running Elastic Stack (use you self hosted Elastic or create a free 14-day trial on Elastic Cloud). This article doesn’t work with AWS managed elasticsearch.
EC2 machine to run Metricbeat agent

IAM policy:

Metricbeat requires certain IAM Policy permissions for it to fetch data from the required resources. An IAM policy is an entity that defines permissions to an object within your AWS environment. Create a customized IAM policy for Metricbeat with specific permissions is needed. Please see Creating IAM Policies for more details. After Metricbeat IAM policy is created, you need to add this policy to the IAM user which provided the credentials in the previous step.

The following table shows the IAM Policies that needs to be added to each Metricbeat:

Consolidated IAM Policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeRegions",
                "ec2:DescribeInstances"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sts:GetCallerIdentity"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:ListAccountAliases"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "tag:getResources"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "rds:DescribeDBInstances",
                "rds:ListTagsForResource"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sns:ListTopics"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:ListQueues"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ce:GetCostAndUsage"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Metricbeat Agent:

Installing the Metricbeat Agent on EC2 machine:

Launch a EC2 machine that runs Ubuntu and login to that machine and run below command to download and install metricbeat.

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.11.1-amd64.deb
sudo dpkg -i metricbeat-7.11.1-amd64.deb

Edit the configuration:

Elastic Cloud:

Modify /etc/metricbeat/metricbeat.yml to set the connection information for Elastic Cloud:

cloud.id: <Get your cloud_id from the Elastic Cloud>
cloud.auth: "elastic:<password>"

Enable the AWS module:

In the out-of-box configuration of Metricbeat, only the system module is enabled by default, so you will need to explicitly enable the AWS module. The following command enables the AWS configuration in the modules.d directory on MacOS and Linux systems:

metricbeat modules enable aws