Configuring CloudWatch Logs Agents on AWS EC2 Instances for Log Analysis: A Step-by-Step Guide Using AWS CLI

Sure, let’s improve the previous article by explaining the AWS CLI parameters and providing examples for each step:

1. SSH into the EC2 Instances:

• ssh: The SSH command allows you to connect to remote servers.
• -i <your_key_pair.pem>: This specifies the path to your private key pair for authentication.
• ec2-user@<instance_ip>: Replace <instance_ip> with the actual IP address of the EC2 instance. This is the SSH username and instance IP. Example:

   ssh -i your-key-pair.pem ec2-user@1.2.3.4

2. Install the CloudWatch Logs Agent:

• sudo yum update -y: This command updates the package repository on the instance.
• sudo yum install -y awslogs: It installs the CloudWatch Logs agent.

3. Configure the CloudWatch Logs Agent:

• sudo nano /etc/awslogs/awslogs.conf: This opens the AWS CloudWatch Logs agent configuration file for editing.
• [syslog]: This is a log group name for the agent’s configuration.
• datetime_format: Specifies the date and time format.
• file: Specifies the log file you want to monitor.
• buffer_duration: Sets the buffer duration.
• log_stream_name: Defines the log stream name.
• initial_position: Specifies where to start reading logs.
• log_group_name: Defines the log group name. Example:

   [syslog]
   datetime_format = %b %d %H:%M:%S
   file = /var/log/messages
   buffer_duration = 5000
   log_stream_name = {instance_id}
   initial_position = start_of_file
   log_group_name = my-ec2-logs

4. Start the CloudWatch Logs Agent:

• sudo systemctl start awslogsd: Starts the CloudWatch Logs agent.
• sudo systemctl enable awslogsd: Enables the agent to start on boot.

5. Analyze Logs with CloudWatch Logs Insights:

• aws logs start-query: Initiates a query in CloudWatch Logs Insights.
• --log-group-name: Specifies the log group name you want to query.
• --start-time and --end-time: Define the time window for your query.
• --query-string: Input the query string.
• --output text: Requests output in text format.
• --query-id: Assigns a query ID. Example:

   aws logs start-query \
     --log-group-name my-ec2-logs \
     --start-time $(($(date +%s)-3600)) \
     --end-time $(date +%s) \
     --query-string "fields @timestamp, @message | sort @timestamp desc | limit 20"

6. Store Results in an S3 Bucket:

• aws s3 cp: Copies a local file to an S3 bucket.
• <results_file>: Specifies the file you want to copy.
• s3://<your-s3-bucket>/<path>: Specifies the S3 bucket and path where you want to store the results. Example:

   aws s3 cp query-results.txt s3://my-bucket/query-results/

By understanding and using these AWS CLI parameters, you can effectively set up and configure a unified CloudWatch Logs agent on your EC2 instances, analyze logs with CloudWatch Logs Insights, and store the results in an S3 bucket.