CloudTrail is a powerful tool for monitoring and auditing AWS deployments, but as a relatively new service, introduced in late 2013, many AWS users may not be aware of its capabilities and potential. As we summarized in this article on AWS logging tools, “CloudTrail records all AWS API calls, making it useful for monitoring access to the management console, CLI usage and programmatic access to other Amazon services. CloudTrail also provides key input for security audits, recording all administrator activity such as policy changes on an S3 bucket, starts and stops on Amazon Elastic Compute Cloud (EC2) instances and changes to user groups or roles.” CloudTrail records data to an S3 bucket in JSON format to facilitate parsing, filtering and data analysis, can trigger alerts via the Simple Notification Service (SNS), can be accessed via custom applications using APIs and feed other logging and operational analysis systems like AWS CloudWatch, Alert Logic, Loggly and Splunk. As we discussed in our earlier article, having a detailed record of API calls is useful for troubleshooting, security forensics and policy compliance audits. Here’s how to get started.
Like all AWS services, CloudTrail is setup and configured using the Web management console or command line interface (CLI). Configuring the service primarily entails specifying an S3 bucket storing the logs, by default the UI will create a new bucket or you can select an existing one, and a couple options. Once enabled, CloudTrail starts recording events which can be viewed via the management console and programmatically queried using the LookupEvents API. You can optionally create an SNS topic that receives notifications when a new log file has arrived.
CloudTrail stores log files in a gzip archive using a standard, hierarchical naming scheme organized by day making it easy to pull entries for specific time periods or individual entries. Log entries can be retrieved using any S3 access method: console, CLI or API. As mentioned, entries are written in JSON format to simplify post-processing or can be viewed directly in the browser via an add-on extension like JSON View. JSON format also allows third-party log analysis tools to aggregate, parse and analyze CloudTrail data.
Configuring CloudTrail with SNS allows users to subscribe to a particular log and be notified whenever it is updated, however topic subscriptions are still managed through the SNS console or API. Since some log files can be quite active, be sure to heed this tip from the CloudTrail documentation:
“Because CloudTrail sends a notification each time a log file is written to the Amazon S3 bucket, an account that’s very active can generate a large number of notifications. If you subscribe using email or SMS, you can end up receiving a large volume of messages. We recommend that you subscribe using Amazon Simple Queue Service (Amazon SQS), which lets you handle notifications programmatically.”
Permissions and Access Controls
Access to logs and other resources CloudTrail uses, like SNS topics, S3 buckets, message queues, etc. is managed through the AWS Identity and Access Management (IAM) system. IAM allows complete control over who can create, configure, or delete CloudTrail entries, start and stop logging, and access the buckets that contain log information. More details about IAM are available in this SearchAWS article, however the IAM policy generator provides an easy interface for creating and editing CloudTrail permissions, including templates for full and read-only access. As per IAM best practices, it’s wise to first create IAM groups like Administrators and Viewers and then add users to the appropriate group. You can also create custom policies using the IAM JSON syntax for special situations, for example read CloudTrail logs and objects in the associated S3 bucket, but not create, update or delete them.
Using CloudTrail with Other Services
CloudTrail’s standard log format and API means it can feed third-party log analysis tools or a custom developed application. One example from the AWS Security blog illustrates how to use CloudTrail, AWS Lambda (an event-triggered compute service) and SNS to generate email notifications when certain APIs in your AWS infrastructure are used. In this scenario, Lambda watches the CloudTrail S3 bucket and triggers an SNS notification when specified APIs are logged. SNS then sends a message to every topic subscriber via email, SMS or mobile push.
Popular log management and analysis products can also consume CloudTrail logs, combining it with data from other AWS services like Config or OpsWorks and on-premise infrastructure to produce comprehensive usage and security reports. Tracking changes across services and infrastructure allows a product like DataDog to correlate change events with performance metrics to help identify the cause of any degradation or highlight the source of any security incidents.
CloudTrail works with every major AWS service, with support for new products regularly being added (the full list is available here). The only charge for using CloudTrail is the S3 storage, which AWS estimates to be less than $3 per account for most customers. Given how easy it is to setup and the availability of free open source log analysis software like Graylog2, using CloudTrail on your AWS infrastructure is a no-brainer.
For more information on using Splunk to analyze AWS data see this datasheet.