Configuring CloudWatch Logs Agents on AWS EC2 Instances for Log Analysis: A Step-by-Step Guide Using AWS CLI

Sure, let’s improve the previous article by explaining the AWS CLI parameters and providing examples for each step:

1. SSH into the EC2 Instances:

  • ssh: The SSH command allows you to connect to remote servers.
  • -i <your_key_pair.pem>: This specifies the path to your private key pair for authentication.
  • ec2-user@<instance_ip>: Replace <instance_ip> with the actual IP address of the EC2 instance. This is the SSH username and instance IP. Example:
   ssh -i your-key-pair.pem ec2-user@

2. Install the CloudWatch Logs Agent:

  • sudo yum update -y: This command updates the package repository on the instance.
  • sudo yum install -y awslogs: It installs the CloudWatch Logs agent.

3. Configure the CloudWatch Logs Agent:

  • sudo nano /etc/awslogs/awslogs.conf: This opens the AWS CloudWatch Logs agent configuration file for editing.
  • [syslog]: This is a log group name for the agent’s configuration.
  • datetime_format: Specifies the date and time format.
  • file: Specifies the log file you want to monitor.
  • buffer_duration: Sets the buffer duration.
  • log_stream_name: Defines the log stream name.
  • initial_position: Specifies where to start reading logs.
  • log_group_name: Defines the log group name. Example:
   datetime_format = %b %d %H:%M:%S
   file = /var/log/messages
   buffer_duration = 5000
   log_stream_name = {instance_id}
   initial_position = start_of_file
   log_group_name = my-ec2-logs

4. Start the CloudWatch Logs Agent:

  • sudo systemctl start awslogsd: Starts the CloudWatch Logs agent.
  • sudo systemctl enable awslogsd: Enables the agent to start on boot.

5. Analyze Logs with CloudWatch Logs Insights:

  • aws logs start-query: Initiates a query in CloudWatch Logs Insights.
  • --log-group-name: Specifies the log group name you want to query.
  • --start-time and --end-time: Define the time window for your query.
  • --query-string: Input the query string.
  • --output text: Requests output in text format.
  • --query-id: Assigns a query ID. Example:
   aws logs start-query \
     --log-group-name my-ec2-logs \
     --start-time $(($(date +%s)-3600)) \
     --end-time $(date +%s) \
     --query-string "fields @timestamp, @message | sort @timestamp desc | limit 20"

6. Store Results in an S3 Bucket:

  • aws s3 cp: Copies a local file to an S3 bucket.
  • <results_file>: Specifies the file you want to copy.
  • s3://<your-s3-bucket>/<path>: Specifies the S3 bucket and path where you want to store the results. Example:
   aws s3 cp query-results.txt s3://my-bucket/query-results/

By understanding and using these AWS CLI parameters, you can effectively set up and configure a unified CloudWatch Logs agent on your EC2 instances, analyze logs with CloudWatch Logs Insights, and store the results in an S3 bucket.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top