StopLogging in CloudTrail: When Someone Turns the Audit Trail Off

What it records

StopLogging suspends recording and log delivery for one trail. AWS describes it as the only way to stop recording and notes that under most circumstances there is no need to use it, because a trail can be updated without stopping. CloudTrail records the call itself, with the trail in requestParameters.name, so the record marks the start of a gap: the trail records nothing more until a StartLogging call names it again.

For a trail enabled in all Regions, the call must come from the trail’s home Region, otherwise it fails with InvalidHomeRegionException, and it cannot be run against the replicated shadow trails in other Regions. The successful record is therefore in the home Region. Other trails, organization trails and the 90 days of management events in Event history are not affected.

When it matters

Turning the trail off is how an intruder makes the next steps invisible: stop logging, do the noisy part (create users, copy data, change policies), then restart logging to make the gap look like an outage. It is MITRE ATT&CK T1562.008 (disable or modify cloud logs), and outside a documented change every successful StopLogging is an incident.

• Any success outside a change window, by any identity.
• A failed attempt. AccessDenied or InvalidHomeRegionException still shows an identity trying to blind you.
• The neighbours. DeleteTrail, UpdateTrail pointing logs at another bucket, PutEventSelectors that log less, and DeleteDetector in GuardDuty around the same time from the same identity are one campaign.

How to investigate

1. Record who stopped which trail, from where and with which key (userIdentity, requestParameters.name, sourceIPAddress, userAgent).
2. Measure the gap: from this eventTime to the next successful StartLogging for the same trail.
3. Fill the gap from sources the attacker did not stop: Event history for management events, an organization trail or another trail, GuardDuty findings, VPC flow logs and service access logs.
4. Read what the same key did before the stop. Preparation such as GetCallerIdentity, role assumptions and permission checks often comes minutes earlier.
5. Restart the trail, deactivate the key or session, and restrict cloudtrail:StopLogging with a service control policy.

Every change to trails in Microsoft Sentinel, oldest first:

AWSCloudTrail
| where TimeGenerated > ago(30d)
| where EventName in ("StopLogging", "StartLogging", "DeleteTrail", "UpdateTrail", "PutEventSelectors")
| extend Request = parse_json(RequestParameters)
| extend Trail = coalesce(tostring(Request.name), tostring(Request.trailName))
| project TimeGenerated, EventName, Trail, UserIdentityArn, UserIdentityType, SourceIpAddress, UserAgent, ErrorCode, AWSRegion
| order by TimeGenerated asc

How long each trail stayed stopped:

AWSCloudTrail
| where TimeGenerated > ago(30d)
| where EventName in ("StopLogging", "StartLogging") and isempty(ErrorCode)
| extend Trail = tostring(parse_json(RequestParameters).name)
| order by Trail asc, TimeGenerated asc
| extend NextEvent = next(EventName), NextTime = next(TimeGenerated), NextTrail = next(Trail)
| where EventName == "StopLogging"
| extend RestartedAt = iff(NextEvent == "StartLogging" and NextTrail == Trail, NextTime, datetime(null))
| project StoppedAt = TimeGenerated, Trail, StoppedBy = UserIdentityArn, SourceIpAddress, RestartedAt,
    MinutesStopped = datetime_diff('minute', coalesce(RestartedAt, now()), TimeGenerated)

In Athena, over a table on the trail’s S3 bucket (named cloudtrail_logs here; a stopped trail delivers nothing new, so run this on another trail’s logs too):

SELECT eventtime, eventname, json_extract_scalar(requestparameters, '$.name') AS trail,
       useridentity.arn, sourceipaddress, errorcode, awsregion
FROM cloudtrail_logs
WHERE eventname IN ('StopLogging', 'StartLogging')
ORDER BY eventtime;