Varun Subramanian
Mar 23, 20217 min
Updated: Mar 24, 2021
Over the course of years the number of cloud native services provided by the cloud platforms have been ever increasing. The platforms do come in with their inbuilt monitoring services but from an administrative standpoint of view it is not that helpful as it does not come with a consolidated view for the end users. To overcome this and to provide a common platform for various service logs and its corresponding visualisation many third party providers have come into the play such as Elastic, Datadog, Splunk etc.
In this article we will be seeing the use of one such service provider: Elastic.
Elastic provides out of the box dashboards to visualise AWS managed services such as Loadbalancers, VPC, S3, CloudTrail and Billing. We will be using a sample Web application hosted on AWS cloud to provide how consolidated dashboards can be viewed in Kibana.
Once we complete the exercise, below are the set of Dashboards that would be available for the Infrastructure Monitoring team to visualise how AWS services are functioning, identify and troubleshoot problems pertaining to any of the services.
Below is a typical web application deployed on AWS cloud that uses a combination of cloud native services and web application server to serve APIs. We also have a static site hosted on Cloudfront to serve static html/css content.
Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics (cpu usage, Disk I/O, network bytes in/out ) from the operating system and from services running on the server. Metricbeat takes the metrics and statistics that it collects and ships them to the output that you specify, such as Elasticsearch or Logstash. Metricbeat by default supports various pre-built modules but in this article we will be focusing on the AWS module.
Metricbeat collects two broad categories of metrics i.e. host metrics and managed services metrics. We will be dedicating one EC2 machine to run metricbeat for AWS managed services metrics collection and another metricbeat on EC2 machine to demonstrate metrics collected from the host server.
NOTE: We will be focusing on metricbeat agent from 7.4 and later
The AWS module for metricbeat currently supports out of the box collection of metrics for the following AWS Services:
EC2
Elastic Loadbalancer
Lambda functions
NAT Gateway
RDS
S3 Storage
SNS
SQS
Transit Gateway
VPN
Billing
Before we dive into installing and configuring metricbeat, let's understand how metricbeat collects, stores and sends metrics to Elastic.
Enable Cloudwatch metrics for each of the AWS Managed services. This will start sending AWS services to CloudWatch, from where Metricbeat will be reading. This documentation gives step by step instructions on how to enable Cloudwatch for a specific AWS service.
Metricbeat will be running on a EC2 machines and configured to collect AWS managed services metrics. This is as simple as enabling the AWS module in metricbeat config file.
Hands-off, Metricbeat periodically queries AWS Cloudwatch to read metrics and sends it to Elasticsearch server, where it is indexed and visualised in Kibana.
Running a Metricbeat requires the following:
AWS account with credentials. Below section covers the Roles required for Metricbeat to read metrics.
Running Elastic Stack (use you self hosted Elastic or create a free 14-day trial on Elastic Cloud). This article doesn’t work with AWS managed elasticsearch.
EC2 machine to run Metricbeat agent
Metricbeat requires certain IAM Policy permissions for it to fetch data from the required resources. An IAM policy is an entity that defines permissions to an object within your AWS environment. Create a customized IAM policy for Metricbeat with specific permissions is needed. Please see Creating IAM Policies for more details. After Metricbeat IAM policy is created, you need to add this policy to the IAM user which provided the credentials in the previous step.
The following table shows the IAM Policies that needs to be added to each Metricbeat:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:DescribeRegions",
"ec2:DescribeInstances"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData",
"cloudwatch:ListMetrics"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"sts:GetCallerIdentity"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"iam:ListAccountAliases"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"tag:getResources"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"rds:DescribeDBInstances",
"rds:ListTagsForResource"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"sns:ListTopics"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"sqs:ListQueues"
],
"Resource": [
"*"
]
},
{
"Effect": "Allow",
"Action": [
"ce:GetCostAndUsage"
],
"Resource": [
"*"
]
}
]
}
Launch a EC2 machine that runs Ubuntu and login to that machine and run below command to download and install metricbeat.
curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.11.1-amd64.deb
sudo dpkg -i metricbeat-7.11.1-amd64.deb
Modify /etc/metricbeat/metricbeat.yml to set the connection information for Elastic Cloud:
cloud.id: <Get your cloud_id from the Elastic Cloud>
cloud.auth: "elastic:<password>"
In the out-of-box configuration of Metricbeat, only the system module is enabled by default, so you will need to explicitly enable the AWS module. The following command enables the AWS configuration in the modules.d directory on MacOS and Linux systems:
metricbeat modules enable aws
To configure AWS credentials, users can put the credentials into the Metricbeat module configuration or use environment variables to pass them. The ability to load AWS credentials from a shared credentials file is added into aws module. Create a shared credential by following these steps. File will be created under the below location.
For Windows:
C:\Users\<yourusername>\.aws\credentials
For Linux, MacOS, or Unix:
~/.aws/credentials
(Optional) Users can specify the profile name using parameter credential_profile_name in aws module config. For more details on AWS credentials types and supported formats, please see AWS credentials configuration for more detail.
metricbeat.modules:
- module: aws
period: 300s
metricsets:
- ec2
- module: aws
period: 300s
metricsets:
- sqs
regions:
- us-west-1
- module: aws
period: 86400s
metricsets:
- s3_request
- s3_daily_storage
- module: aws
period: 300s
metricsets:
- cloudwatch
metrics:
- namespace: AWS/EC2
name: ["CPUUtilization"]
statistic: ["Average"]
- namespace: AWS/EBS
- namespace: AWS/ELB
resource_type: elasticloadbalancing
- module: aws
period: 60s
metricsets:
- elb
- natgateway
- rds
- transitgateway
- usage
- vpn
namespace: A namespace in AWS CloudWatch is a container for metrics from a specific application or service. Each service has its own namespace, for example Amazon EC2 uses AWS/EC2 namespace and Amazon Elastic Block Storage uses AWS/EBS namespace. Please see the full list of services and namespaces that publish CloudWatch metrics for more details.
name: Users can specify what are the specific CloudWatch metrics that need to be collected per namespace.
dimensions: Dimensions are used to refine metrics returned for each instance. For example, InstanceId, ImageId and InstanceType all can be used as dimensions to filter data requested from Amazon EC2.
statistic: Users can specify one or more statistic methods for each CloudWatch metric setting. By default, average, count, maximum, minimum and sum will all be collected for each metric.
tags: resource_type_filter: Tags for resources will not be collected unless this parameter is set. Each resource in AWS has a specific resource type and the common format is service[:resourceType]. Please see resource type filters for more details.
credential_profile_name: If the aws credentials config is done it will be automatically picked up, else one can enter the details manually.
The setup command loads the Kibana dashboards. If the dashboards are already set up, omit this command.
sudo metricbeat setup
sudo service metricbeat start
Metricbeat comes with pre-built Kibana dashboards and UIs for visualizing log data. The dashboards would have been loaded earlier when the setup command was run.
In the Kibana side navigation:
Click Discover, to see Metricbeat data. Also make sure the predefined metricbeat-* index pattern is selected.
Click Dashboard, then search for "AWS metric*", select the dashboard that you want to open.
Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing. Filbeat by default supports various pre-built modules but in this article we will be focusing on the AWS module.
The AWS module for filebeat currently supports the following:
CloudTrail
CloudWatch
EC2
Elastic Loadbalancer
S3 Access
VPC Flow logs
Running a filebeat requires the following:
Configure AWS Services to send logs to S3. All files can be placed on the same bucket.
Setting up the SQS service in the AWS Account. (SQS Service). This SQS service will be used to notify filebeat when new file is placed in S3 bucket configured in step 1.
AWS account with credentials.
Running Elastic Stack. (This can either be a cluster in Elasticsearch Service on Elastic Cloud)
Filebeat agent
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.11.1-amd64.deb
sudo dpkg -i filebeat-7.11.1-amd64.deb
1. Modify /etc/filebeat/filebeat.yml to set the connection information for Elastic Cloud:
cloud.id: <Get your cloud_id from the Elastic Cloud>
cloud.auth: "elastic:<password>"
2. Enable S3 to send a notification to SQS when a new file is placed in bucket.
Steps to Enable S3 Event notification to SQS are documented here.
From the configuration of Filebeat, one will need to explicitly enable the AWS module. The following command enables the AWS configuration in the modules.d directory on MacOS and Linux systems:
sudo filebeat modules enable aws
(Skip, if this has already been done during metricbeat setup)
To configure AWS credentials, users can put the credentials into the Filebeat module configuration or use environment variables to pass them. Create a shared credential by following these steps. File will be created under the below location.
For Windows:
C:\Users\<yourusername>\.aws\credentials
For Linux, MacOS, or Unix:
~/.aws/credentials
(Optional) Users can specify the profile name using parameter credential_profile_name in aws module config. For more details on AWS credentials types and supported formats, please see AWS credentials configuration for more detail.
- module: aws
cloudtrail:
enabled: true
var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
cloudwatch:
enabled: true
var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
ec2:
enabled: true
var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
elb:
enabled: true
var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
s3access:
enabled: true
var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
vpcflow:
enabled: true
var.queue_url: https://sqs.myregion.amazonaws.com/123456/myqueue
var.queue_url: (Required) AWS SQS queue url.
The setup command loads the Kibana dashboards. If the dashboards are already set up, omit this command.
sudo filebeat setup
sudo service filebeat start
Filebeat comes with pre-built Kibana dashboards and UIs for visualizing log data. The dashboards would have been loaded earlier when the setup command was run.
In the Kibana side navigation:
Click Discover, to see Filebeat data. Also make sure the predefined filebeat-* index pattern is selected.
Click Dashboard, then search for "AWS Filebeat*", select the dashboard that you want to open.