Manipulating data with S3 is as simple as uploading a file, or it can be done through an easy-to-construct API call. But this simplicity can come with a problem. Let’s consider the potential situation wherein you realize that the data in your S3 bucket is not quite right. Perhaps files are missing, files have been modified when they shouldn’t have been, or some suspect files have started to appear. This need not be the result of nefarious activity—It may simply be the result of a microservice mishandling data due to a software bug.
To detect and address issues with your S3 data, you need a way to monitor and audit calls to the S3 API. That is the subject of this post.
Logging S3 API Calls and Tracking Changes with CloudTrail
Monitoring API calls wasn’t always easy, at least not before the introduction in late 2013 of AWS CloudTrail. Essentially, CloudTrail is an AWS Service which tracks calls to the APIs in your account, keeping track of:
- Time of the API call
- Identity of the caller, including the IP address
- Request parameters
- The resulting response
In order to enable CloudTrail on your S3 API calls, log into your AWS Management Console and navigate to the AWS CloudTrail home page. Alternately, you can simply append /cloudtrail/home to the URL for your AWS console. The resultant URL should look similar to the following, depending on which region you are in:
https://us-west-2.console.aws.amazon.com/cloudtrail/home
Click on the Get Started Now button. Select a name for the CloudTrail service, and determine if you want to enable this service for all regions. Then determine if you want to monitor an existing S3 bucket, or if you want to create a new bucket, and choose the appropriate option. If you intend to monitor an existing bucket, select it from the available option next to the S3 bucket label. Other options are available, but this will give you visibility into basic operations on the selected bucket.
Select Turn On
Data is great, but I want to know the instant things go awry
With a trail enabled on your S3 bucket, any actions through the console, or through API calls to that bucket, are going to be logged. Upload a new file, or perform another action on your “trailed” S3 bucket and then return to the CloudTrail home page. You should now see a listing of events detailing the actions you just performed. You can click to view more information on each.
This is really handy if you know what you’re looking for, but you’ll probably want to know exactly when things start performing contrary to what you expect.
In order to do that, you’ll want to feed your CloudTrail logs into CloudWatch. From CloudWatch you can set up alarms, track metrics and monitor trends and maintain a view of activity in your S3 buckets in addition to your other services in the AWS ecosystem.
Configuring CloudTrail to Send Logs to CloudWatch
Return to the CloudTrail home page, and click on the name of your trail. This will bring you to the configuration page for the trail. Scroll to the bottom of the page, and click on the Configure button under the CloudWatch Logs header.
First, you’ll want to select either an existing Log Group to send the logs to, or create a new one. Specify either the name of the existing group or enter a new group name, and click Continue.
We will now need to create a role to allow the interaction between CloudTrail and CloudWatch. AWS will walk you through this process. Click on the View Details link. By default, the IAM role should be set to CloudTrail_CloudWatchLogs. If you want to use a different IAM role, you can specify it here.
If you have an existing policy you would like to use, you can select it next to Policy Name, or you can select Create a New Role Policy in that same dropbox. Finally, click on Allow to finish the process and return yourself to the CloudTrail configuration screen. You should now see the configuration for the CloudWatch logs under the section of the same name, and all CloudTrail logs should be included in your CloudWatch logs.
Further reading
For more information on setting up alarms and general use of CloudWatch, check out the CloudWatch user guide.
About the Author
Mike Mackrory is a Global citizen who has settled down in the Pacific Northwest – for now. By day he works as a Senior Engineer on a Quality Engineering team and by night he writes, consults on several web based projects and runs a marginally successful eBay sticker business. When he’s not tapping on the keys, he can be found hiking, fishing and exploring both the urban and the rural landscape with his kids. Always happy to help out another developer, he has a definite preference for helping those who bring gifts of gourmet donuts, craft beer and/or Single-malt Scotch.
