Hands-on:Implementing CloudWatch Metrics and Alarms with AWS CLI for Automated Monitoring – Kodecamps- Learn to code in a fun and interactive way

To implement the solution using the AWS Command Line Interface (CLI), follow these steps to download and install the Amazon CloudWatch agent on your on-premises servers, send logs to Amazon CloudWatch Logs, create a metric filter, and set up a CloudWatch Alarm. Please note that this process involves configuring the CloudWatch agent on your on-premises servers, and the specific installation steps may vary based on your operating system.

Step 1: Download and Install the CloudWatch Agent

Download the Amazon CloudWatch agent from the official AWS documentation: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html
Install the agent on your on-premises servers. The installation process will depend on your server’s operating system. Here are some common commands for Linux and Windows:

For Linux:

sudo rpm -Uvh https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start

For Windows:

Download the CloudWatch agent MSI installer from the provided link and run the installer.

Step 2: Configure the CloudWatch Agent

Create a configuration file that defines what logs to collect and where to send them. You can create a JSON or TOML configuration file. Here’s an example JSON configuration:

{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "root"
    },
    "metrics": {
        "append_dimensions": {
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
            "ImageId": "${aws:ImageId}",
            "InstanceId": "${aws:InstanceId}",
            "InstanceType": "${aws:InstanceType}"
        },
        "metrics_collected": {
            "disk": {
                "measurement": [
                    "used_percent"
                ],
                "resources": [
                    "*"
                ]
            },
            "mem": {
                "measurement": [
                    "mem_used_percent"
                ]
            }
        }
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/syslog",
                        "log_group_name": "syslog",
                        "log_stream_name": "{instance_id}"
                    }
                ]
            }
        },
        "log_stream_name": "{instance_id}",
        "force_flush_interval": 15
    }
}

Use the amazon-cloudwatch-agent-ctl command to configure the agent with the created configuration file. For example:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -s -c /opt/aws/amazon-cloudwatch-agent/bin/config.json

Step 3: Create a Metric Filter

Open the Amazon CloudWatch console.
In the navigation pane, choose “Log groups.”
Choose the log group where the logs are being sent (e.g., “syslog”).
Choose “Create Metric Filter” and configure it to extract numeric data from the logs, typically using a filter pattern.

Creating a metric filter in Amazon CloudWatch using the AWS CLI requires several steps. You’ll need to define the metric filter and its filter pattern, associate it with a log group, and then publish the filter. Here’s how you can implement this using the AWS CLI:

Step 1: Define a Metric Filter

First, define the metric filter by creating a JSON or YAML file that contains the filter pattern and other filter details. For example, create a file named metric-filter.json with the following content:

{
  "filterName": "MyMetricFilter",
  "filterPattern": "[timestamp, field1=value1, field2=value2]",
  "logGroupName": "your-log-group-name",
  "metricTransformations": [
    {
      "metricName": "MyCustomMetric",
      "metricNamespace": "CustomNamespace",
      "metricValue": "1",
      "defaultValue": 0
    }
  ]
}

filterName: A unique name for your metric filter.
filterPattern: The filter pattern to extract numeric data. This pattern should match the log entries that contain the data you want to convert to metrics.
logGroupName: The name of the log group where you want to apply this metric filter.
metricTransformations: Define the transformation details for the extracted metric data. In this example, it creates a custom metric named “MyCustomMetric” in the “CustomNamespace.”

Step 2: Create the Metric Filter

Use the AWS CLI to create the metric filter by providing the metric-filter.json file as input:

aws logs put-metric-filter --cli-input-json file://metric-filter.json

Replace "your-log-group-name" with the actual name of your log group.

Step 3: Publish the Metric Filter

Publish the metric filter by using the following command:

aws logs test-metric-filter --filter-name MyMetricFilter --log-event-messages "[timestamp, field1=value1, field2=value2]" --log-group-name your-log-group-name

Replace "your-log-group-name" with the actual name of your log group.

This command will test the metric filter by providing a sample log event message that matches the filter pattern. If the filter pattern successfully matches the log event, it will extract the metric and publish it to CloudWatch.

After completing these steps, you should have a metric filter configured to extract numeric data from your logs and publish them as CloudWatch custom metrics. Make sure to customize the filter pattern, metric name, and other details according to your specific log data and requirements.

Step 4: Create a CloudWatch Alarm

In the Amazon CloudWatch console, navigate to “Alarms.”
Choose “Create Alarm.”
Select the metric filter that you’ve created.
Configure the alarm actions, such as sending a notification to the IT Operations team.
Complete the alarm creation process.

Creating a CloudWatch Alarm using the AWS CLI involves a few steps. You need to define the alarm, set up alarm actions, and create the alarm itself. Here’s how you can implement this using the AWS CLI:

Step 1: Define the Alarm Configuration

You’ll first need to create a JSON or YAML file that contains the alarm configuration. For example, create a file named cloudwatch-alarm.json with the following content:

{
  "AlarmName": "MyCustomAlarm",
  "AlarmDescription": "This is my custom CloudWatch alarm",
  "ComparisonOperator": "GreaterThanThreshold",
  "EvaluationPeriods": 1,
  "MetricName": "MyCustomMetric",
  "Namespace": "CustomNamespace",
  "Period": 60,
  "Statistic": "SampleCount",
  "Threshold": 1,
  "TreatMissingData": "missing",
  "ActionsEnabled": true,
  "AlarmActions": ["arn:aws:sns:us-east-1:123456789012:MySNSTopic"],
  "OKActions": ["arn:aws:sns:us-east-1:123456789012:MySNSTopic"],
  "AlarmDescription": "My custom CloudWatch alarm",
  "Dimensions": [
    {
      "Name": "LogGroupName",
      "Value": "your-log-group-name"
    }
  ]
}

AlarmName: A unique name for your alarm.
ComparisonOperator: The comparison operator for the alarm.
EvaluationPeriods: The number of periods to evaluate the alarm.
MetricName and Namespace: The metric name and namespace should match what you defined in your metric filter.
Period: The period (in seconds) over which the alarm should be checked.
Threshold: The threshold value that triggers the alarm.
ActionsEnabled: Set to true to enable alarm actions.
AlarmActions and OKActions: Specify the ARNs of SNS topics or other actions to be triggered when the alarm state changes.
Dimensions: Define dimensions that match the log group name.

Step 2: Create the CloudWatch Alarm

Use the AWS CLI to create the CloudWatch alarm by providing the cloudwatch-alarm.json file as input:

aws cloudwatch put-metric-alarm --cli-input-json file://cloudwatch-alarm.json

Replace "arn:aws:sns:us-east-1:123456789012:MySNSTopic" with the actual ARN of the SNS topic you want to use for alarm notifications.

Step 3: Complete the Alarm Creation

After running the command to create the alarm, it will be created in your AWS account. You can now complete the alarm creation process.

Your CloudWatch alarm is now configured to monitor the metric produced by your metric filter, and it will take the defined actions when the alarm state changes based on the specified threshold. Be sure to adjust the alarm settings and actions according to your specific requirements.

With these steps, you’ve successfully implemented the solution to collect logs from your on-premises servers, create metric filters to extract numerical metrics from the logs, and set up CloudWatch alarms to monitor these metrics and notify your IT Operations team in case of issues. Make sure to customize the configuration and alarm thresholds according to your specific requirements.

Leave a Comment Cancel Reply