CloudTrail logs every API call made in your AWS account — who did what, when, from where. It’s foundational for security incident response, compliance evidence, and understanding account activity. It’s also partially configured in most AWS accounts I’ve audited, which means the logs are there when nothing is wrong and missing precisely when something is.
Here’s what a complete CloudTrail configuration looks like.
What CloudTrail captures
CloudTrail records three event categories:
Management events — Control plane operations: creating EC2 instances, modifying IAM policies, changing S3 bucket policies, enabling/disabling services. These are on by default for new CloudTrail trails. Free for first copy per region.
Data events — Data plane operations: S3 object reads and writes, Lambda function invocations, DynamoDB PutItem/GetItem, RDS query execution. Off by default. Priced at $0.10 per 100,000 events. High volume in active accounts.
Insights events — Anomaly detection on unusual API call rates or error rates. Priced at $0.35 per 100,000 events analyzed. Useful for detecting credential abuse and automated attack patterns.
The distinction matters: management events tell you someone created an S3 bucket; data events tell you someone read a specific object from that bucket.
The default CloudTrail problem
AWS enables a partial CloudTrail configuration out of the box in new accounts — specifically, it enables the “Event history” feature that gives you 90 days of management events viewable in the console. This is not the same as a CloudTrail trail.
What Event history gives you:
- 90 days of management events in the current region
- No S3 logging
- No log file validation
- No cross-region coverage
What a proper trail gives you:
- Persistent logging to S3 with configurable retention
- Log file integrity validation (tamper detection)
- All-region coverage with a single trail
- CloudWatch Logs integration for alerting
- Data event logging (optional)
- Organization-level trails (covers all accounts)
If your account has only the default Event history and not a configured trail, you have no persistent audit logs. 90 days of console history is not audit evidence.
Creating a complete trail
A production CloudTrail configuration:
resource "aws_cloudtrail" "main" {
name = "main-trail"
s3_bucket_name = aws_s3_bucket.cloudtrail_logs.id
include_global_service_events = true # IAM, STS, CloudFront
is_multi_region_trail = true # All regions
enable_log_file_validation = true # Tamper detection
cloud_watch_logs_group_arn = "${aws_cloudwatch_log_group.cloudtrail.arn}:*"
cloud_watch_logs_role_arn = aws_iam_role.cloudtrail_cloudwatch.arn
event_selector {
read_write_type = "All"
include_management_events = true
# Data events for S3 — all buckets
data_resource {
type = "AWS::S3::Object"
values = ["arn:aws:s3:::"] # All buckets
}
}
kms_key_id = aws_kms_key.cloudtrail.arn # Encrypt logs at rest
}
Critical settings:
is_multi_region_trail = true — A single-region trail misses activity in other regions. Attackers commonly create resources in regions you don’t monitor. A multi-region trail captures all regions with one configuration.
include_global_service_events = true — IAM, STS, and CloudFront events are global and only appear in us-east-1 by default. This includes them in your trail regardless of trail region.
enable_log_file_validation = true — CloudTrail generates a digest file every hour containing hash values for all log files in that period. If a log file is modified or deleted, validation fails. Required for compliance frameworks that need tamper-evident logs.
kms_key_id — Encrypt log files with a CMK. Without this, logs are stored in S3 with SSE-S3 (AES-256). With a CMK, you control who can decrypt CloudTrail logs and can audit decrypt operations via CloudTrail itself (yes, CloudTrail logs its own decryption calls).
The S3 bucket for log storage
resource "aws_s3_bucket" "cloudtrail_logs" {
bucket = "my-company-cloudtrail-logs"
}
resource "aws_s3_bucket_policy" "cloudtrail_logs" {
bucket = aws_s3_bucket.cloudtrail_logs.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "AWSCloudTrailAclCheck"
Effect = "Allow"
Principal = { Service = "cloudtrail.amazonaws.com" }
Action = "s3:GetBucketAcl"
Resource = aws_s3_bucket.cloudtrail_logs.arn
Condition = {
StringEquals = {
"aws:SourceArn" = "arn:aws:cloudtrail:us-east-1:${data.aws_caller_identity.current.account_id}:trail/main-trail"
}
}
},
{
Sid = "AWSCloudTrailWrite"
Effect = "Allow"
Principal = { Service = "cloudtrail.amazonaws.com" }
Action = "s3:PutObject"
Resource = "${aws_s3_bucket.cloudtrail_logs.arn}/AWSLogs/${data.aws_caller_identity.current.account_id}/*"
Condition = {
StringEquals = {
"s3:x-amz-acl" = "bucket-owner-full-control"
"aws:SourceArn" = "arn:aws:cloudtrail:us-east-1:${data.aws_caller_identity.current.account_id}:trail/main-trail"
}
}
}
]
})
}
Retention: Add an S3 lifecycle rule to match your compliance requirement. PCI DSS and SOC 2 typically require 12 months; HIPAA requires 6 years. Archive to Glacier or Glacier Deep Archive after 90 days to reduce storage costs significantly (Glacier Deep Archive is $0.00099/GB-month vs. S3 Standard at $0.023/GB-month).
Block public access: The CloudTrail log bucket must never be public. Block all public access.
CloudWatch Logs integration for alerting
Shipping CloudTrail to CloudWatch Logs enables metric filters and alarms on specific event patterns.
Essential CloudWatch metric filters for CloudTrail:
Root account usage:
{ $.userIdentity.type = "Root" && $.userIdentity.invokedBy NOT EXISTS && $.eventType != "AwsServiceEvent" }
IAM policy changes:
{ ($.eventName=DeleteGroupPolicy) || ($.eventName=DeleteRolePolicy) || ($.eventName=DeleteUserPolicy) || ($.eventName=PutGroupPolicy) || ($.eventName=PutRolePolicy) || ($.eventName=PutUserPolicy) || ($.eventName=CreatePolicy) || ($.eventName=DeletePolicy) || ($.eventName=CreatePolicyVersion) || ($.eventName=DeletePolicyVersion) || ($.eventName=SetDefaultPolicyVersion) || ($.eventName=AttachRolePolicy) || ($.eventName=DetachRolePolicy) || ($.eventName=AttachUserPolicy) || ($.eventName=DetachUserPolicy) || ($.eventName=AttachGroupPolicy) || ($.eventName=DetachGroupPolicy) }
CloudTrail changes (detecting tampering):
{ ($.eventName=CreateTrail) || ($.eventName=UpdateTrail) || ($.eventName=DeleteTrail) || ($.eventName=StartLogging) || ($.eventName=StopLogging) }
Security group changes:
{ ($.eventName=AuthorizeSecurityGroupIngress) || ($.eventName=AuthorizeSecurityGroupEgress) || ($.eventName=RevokeSecurityGroupIngress) || ($.eventName=RevokeSecurityGroupEgress) || ($.eventName=CreateSecurityGroup) || ($.eventName=DeleteSecurityGroup) }
Console sign-in failures:
{ ($.eventName=ConsoleLogin) && ($.errorMessage="Failed authentication") }
Set CloudWatch alarms on these metric filters with SNS notifications. These are the events that matter for security monitoring — root usage, IAM changes, and CloudTrail modification are the first things an attacker would do after gaining access.
Organization trails
If you have multiple AWS accounts in an AWS Organization, use an organization trail instead of configuring CloudTrail per account:
resource "aws_cloudtrail" "org_trail" {
name = "org-trail"
s3_bucket_name = aws_s3_bucket.cloudtrail_logs.id
is_multi_region_trail = true
include_global_service_events = true
enable_log_file_validation = true
is_organization_trail = true # Covers all accounts in org
}
The organization trail captures events across all current and future member accounts and writes them to a single S3 bucket. Member accounts can see their own events but cannot modify or disable the organization trail. This is the correct pattern for centralized security monitoring.
Required: The trail must be created in the management account (root account of the organization). Member accounts cannot create organization trails.
Data events: what to enable and what to skip
Data events at $0.10/100K events can get expensive. Prioritization:
Always enable:
- S3 data events for sensitive buckets — The buckets containing PII, PHI, payment data, secrets, or backup archives. You need to know who read what from these buckets for incident response and compliance evidence.
- Lambda invocation events for sensitive functions — Functions that handle payment processing, PHI access, or credential management.
Enable selectively:
- S3 data events for all buckets — Appropriate for compliance-sensitive accounts (healthcare, PCI scope). At scale, can generate significant event volume and cost. Consider CloudTrail Lake (below) for cost-efficient querying instead.
- DynamoDB events — Enable for tables containing sensitive data.
Usually skip:
- S3 data events for static assets, log archives, or public buckets — generates high volume, low security value.
CloudTrail Lake
CloudTrail Lake is a managed analytics repository that stores and queries CloudTrail events using SQL. Alternative to S3 + Athena for CloudTrail analysis.
Pricing: $0.005/GB ingested + $0.005/GB scanned for queries. Retention: 7 years maximum.
When CloudTrail Lake makes sense:
- You run frequent ad-hoc investigations into CloudTrail data
- You want a managed solution without S3 + Athena setup
- You need 7-year retention without managing S3 lifecycle policies
When to stick with S3 + Athena:
- You have an existing Athena/Glue setup
- You need to join CloudTrail events with other data sources
- Your query patterns are predictable and can be cost-optimized with partitioning
What compliance frameworks require
SOC 2 Trust Service Criteria (Availability, Security): Requires logging of significant events, monitoring for anomalies, and evidence that logs are reviewed. CloudTrail + CloudWatch alarms + a review process satisfies this. Log retention: typically 12 months minimum.
PCI DSS v4.0 Requirement 10: Requires audit logs for all access to cardholder data, all actions by individuals with root or administrative privileges, all access to audit logs, and all invalid logical access attempts. CloudTrail with data events on CDE S3 buckets and Lambda functions, plus CloudWatch alarms, directly addresses these requirements.
HIPAA: Requires audit controls that record and examine activity in systems that contain PHI. CloudTrail with data events for PHI storage (S3, DynamoDB, RDS via proxy) provides this. Retention: 6 years.
Common gaps in existing configurations
The gaps I find most often in CloudTrail audits:
Single-region trail. Trail is configured in us-east-1, but the account has resources in us-west-2, eu-west-1, or other regions with no coverage.
No log file validation. Logs are shipped to S3 but enable_log_file_validation is false. An attacker who gains S3 access could delete or modify logs undetected.
No CloudWatch Logs integration. Logs go to S3 but no alerting. Root account usage and IAM changes go undetected in real time.
CloudTrail disabled in some accounts. In multi-account setups without an organization trail, individual accounts have disabled or misconfigured their own trails.
S3 bucket is in the same account. CloudTrail logs for account X stored in account X — an attacker who compromises account X can delete the evidence. Logs should go to a dedicated security/log-archive account.
Cross-account log centralization
The secure pattern for multi-account CloudTrail:
Production account → CloudTrail → S3 bucket in dedicated security account
Staging account → CloudTrail → S3 bucket in dedicated security account
Dev account → CloudTrail → S3 bucket in dedicated security account
The security account’s S3 bucket has a bucket policy allowing s3:PutObject from each account’s CloudTrail service principal. No account except the security account has s3:DeleteObject on the log bucket. Even if an account is compromised, the attacker cannot delete their own CloudTrail logs.
Getting this right
CloudTrail configuration is one of the items I always check in AWS security audits — it’s foundational, and the gaps are predictable. If you’re setting up CloudTrail for a compliance requirement or want a review of your current configuration, I’m available to help.
Nick Allevato is an AWS Certified Solutions Architect Professional with 20 years of infrastructure experience. He runs Cold Smoke Consulting, an independent AWS consulting practice.