How to Setup and Push Application Logs to AWS Cloudwatch

AWS Cloudwatch logs service has the capability to store custom logs generated from your application instances. Form web servers either Apache & Nginx access or error logs can be pushed to Cloudwatch logs It acts as central log management for your applications running on AWS.

This article walks you through the steps involved in configuring the Cloudwatch agent on an ec2 instance and configures it to push the desired logs.

Forward Application Logs To Cloudwatch

You can send logs from any number of sources to Cloudwatch. All you need to have is a Cloudwatch agent running on your instance.

Here is what you have to do

1. Create a custom ec2 IAM role with Cloudwatch write access
2. Install Cloudwatch logs ec2 agent
3. Configure log sources in the Cloudwatch agent configuration file.
4. Validate logs in Cloudwatch dashboard.

Let's get started with the setup.

Create an IAM role for Cloudwatch

To set up AWS custom logs, first, you need to create and add an IAM role to your instance. This IAM role will have write access to Cloudwatch service so that all the logs can be shipped to Cloudwatch.

Before creating a role, you need to create a custom policy.

Step 1: Head over to AWS IAM -> Policy -> Create Policy

Step 2: Select the JSON option

Step 3: Copy the following content in the policy block. In the next page, give a name, description for your policy and click create policy option.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*"
    ]
  }
 ]
}

Once you create the policy, you need to create a role with the custom policy you have created.

Step 4: Head over to AWS IAM -> Roles and select options as shown below.

Step 5: From the filter, select “Customer Manager” and select the Policy you created in step 3.

Step 6: Next, enter a role name and create the role.

Add the Cloudwatch Role to the Instance

1. Now, head over to ec2 and select the instance in which you want to configure the custom logs.
2. Right-click for options and select Instance Settings and then choose Attach/Replace IAM Role option.
3. On the next page, select the custom cloud watch IAM role you created from the dropdown and choose to apply.

Setup Cloudwatch Logs Agent

SSH into the ec2 instance and follow the steps given below.

Step 1: Download the official cloudwatch agent setup script

curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O

Step 3: Execute the python script with your AWS region as a parameter.

Few things to understand before configuring the agent.

1. Make sure you have python installed on your server. Also in the latest server versions, by default python3 will be available.
2. When prompted for the access key and secret key, just hit enter without providing any values because we are using custom IAM roles with Cloudwatch write permissions.
3. You can predefine a log-group in Cloudwatch and use the same name while configuring the agent so that all logs will be pushed to that log group.
4. Provide valid log path files and custom names for identifying it in the Cloudwatch dashboard.

sudo python ./awslogs-agent-setup.py --region us-west-1

The above script will ask for log file location other options for managing logs in Cloudwatch.

You can manage the logs agent service using the following command.

sudo service awslogs start
sudo service awslogs stop
sudo service awslogs restart

All the aws logs config files and startup scripts can be found under/var/awslogs folder.

You can add additional log configs in the /var/awslogs/etc/awslogs.conf file. After making the changes, make sure you restart the agent.

Here is an example config for pushing nginx access logs to webserver-logs group.

[/var/log/nginx]
datetime_format = %b %d %H:%M:%S
file = /var/log/nginx/access.log
buffer_duration = 5000
log_stream_name = web-server-01
initial_position = start_of_file
log_group_name = webserver-logs

AWS specific configuration can be edited in /var/awslogs/etc/aws.conf file.

The agent start and other scripts can be found under /var/awslogs/bin folder.

Validating Custom Logs in Cloudwatch Dashboard

Once the setup is done, you can view all the configured logs under cloudwatch dashboard (under logs option)

1. Go to Logs -> Log Groups and you will see the log group you mentioned in the agent configuration.
2. Select the log group and you should see your instance identified you mentioned in the config.
3. If you click the instance identifier, it shows all the logs. You can use the cloud watch filter option to filter and query required logs.

Conclusion

We have explained the Cloudwatch logs agent setup to push application logs to the Cloudwatch logging service. It is a manual setup. If you want this to be automated, all the agent configuration has to be baked in the ec2 AMI. Few configurations can be added at the system startup using the user data scripts. Again, it depends on what workflow you are opting for.