1. Expel Support Center
  2. Connect your technology
  3. Cloud integrations

AWS CloudTrail — Existing CloudTrail setup for Workbench

Sharon Burton
  • Updated

This article describes how to connect your existing AWS CloudTrail to the Expel Workbench.

If you are connecting an existing CloudTrail that includes Control Tower, use the Control Tower connection procedure instead.

If you are connecting a new CloudTrail, use the onboarding wizard or the manual connection procedure instead.

Tip

This article was accurate at the time of writing, but changes happen. If you find the instructions are outdated, leave a description in the comment field below and let us know!

Amazon S3 notifications can’t be configured for more than 1 Simple Queue Service (SQS) queue per bucket. This poses a problem for customers who want to hook up Workbench to their CloudTrail bucket if they're already sending notifications to an SQS queue for internal reasons.

In this article, we show how you can route S3 notifications through SNS (simple notification service) to a different SQS queue that Workbench can poll from.

Before you start

  1. A trail must already be configured to send CloudTrail events to a S3 bucket.

  2. An IAM (Identity and Access Management) Role must be created for Workbench to authenticate with. To set this up, follow Step 6 of the AWS CloudTrail - New CloudTrail setup for Workbench article.

Information to record

Field

Description

S3 bucket ARN (Amazon Resource Name)

The unique identifier for the S3 bucket where CloudTrail events are being sent.

SQS queue ARN

The unique identifier for the SQS queue that Workbench uses to consume S3 notifications.

SQS queue URL

The URL for the SQS queue that Workbench uses to consume S3 notifications.

SNS topic ARN

The unique identifier for the SNS topic that routes S3 notifications to the SQS queue.

Step 1: Create a new SNS topic

Caution

Make sure to create the SNS topic in the same region as the S3 bucket CloudTrail events are being sent to!

  1. Navigate to Services > Simple Notification Service > Topics and click Create Topic.

    Figure_1.png

  2. On the next screen, select Standard as the Type and create a Topic Name.

    Figure_2.png

  3. Under Access Policy, select Advanced.

    Figure_3.png

  4. In the JSON editor, paste the below policy substituting the YOUR_TOPIC_ARN and YOUR_S3_ARN fields with your values. This policy allows S3 to publish notifications to the topic for your CloudTrail bucket.

    {
"Version": "2008-10-17",
"Id": "expel-topic-policy-ID",
"Statement": [
  {
   "Sid": "expel-statement-ID",
   "Effect": "Allow",
   "Principal": {
    "AWS":"*"
    },
    "Action": [
     "SNS:Publish"
    ],
    "Resource": "YOUR_TOPIC_ARN",
    "Condition": {
       "ArnLike": { 
           "aws:SourceArn": "YOUR_S3_ARN"
    }
   }
  }
 ]
}

  5. Click Create Topic.

Step 2: Create an SQS queue for S3 notifications

In this step, we create a new SQS queue for S3 notifications. Workbench polls notifications from this queue to know when new CloudTrail data is added.

Caution

Make sure you create the SQS queue in the same region as the SNS topic and S3 bucket!

  1. Navigate to Services > Simple Queue Service > Queues and click Create queue.

    Figure_4.png

  2. On the next screen, select Standard Queue and name the new queue.

    Figure_5.png

    • Visibility timeout: 30 Seconds.

    • Message retention period: 7 days.

    • Delivery delay: 0 Seconds.

    • Maximum message size: 256 KB.

    • Receive message wait time: 0 Seconds.

  3. Under Access Policy, select Advanced.

    Figure_6.png

  4. In the JSON editor, paste the below policy substituting the YOUR_SQS_QUEUE_ARN and YOUR_SNS_TOPIC_ARN fields with your values.

    {
"Version": "2012-10-17",
"Id": "YOUR_QUEUE_ARN/SQSDefaultPolicy",
"Statement": [
{
"Sid": "Sid1572965666162",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": "SQS:SendMessage",
"Resource": "YOUR_SQS_QUEUE_ARN",
"Condition": {
"StringEquals": {
"aws:SourceArn": "YOUR_SNS_TOPIC_ARN"
}
}
}
]
}

Step 3: Subscribe SQS queue to SNS topic

Now that we’ve created an SNS topic and SQS queue, we need to configure SNS to send events to the SQS queue.

  1. Navigate to Services > Simple Notification Service > Subscriptions and click Create subscription.

  2. On the next screen, type the required fields to configure the subscription.

    Field

    Value

    Topic ARN

    Your SNS Topic ARN

    Protocol

    Select Amazon SQS

    Endpoint

    Your SNS Queue ARN

    Enable raw message delivery

    Select this option

    Important

    Selecting enable raw message delivery makes sure SNS doesn’t add extra metadata headers to the message when it sends to SQS. Make sure you select this!

  3. Click Create subscription to finish this step.

    Screen_Shot_2021-03-05_at_9.39.51_AM.png

Step 4: Enable S3 event notifications to SNS topic

In this step, we configure the CloudTrail S3 bucket to send SNS notifications when CloudTrail adds logs to the bucket.

  1. Navigate to Services > S3 > Your S3 CloudTrail Bucket.

  2. Open Properties for your S3 bucket and navigate to Event notifications. Click Create event notification.

    Figure_8.png

  3. On the next screen:

    • Create a name for your notification rule.

    • Select All object create events.

    • Send to SNS topic.

    • Select your SNS topic we created earlier.

      Figure_9.png
      Figure_10.png

  4. Click Save changes.

Step 5: Grant Expel IAM Role access to SQS queue

At this point we configured S3 notifications → SNS topic → SQS queue. The final step involves granting the existing Expel IAM Role access to poll events from the SQS queue and the S3 bucket.

  1. Navigate to Services > IAM > Expel Role.

  2. Create and add a new inline policy to the Expel Role. The policy should grant the permissions below:

    • SQS Permissions

      • DeleteMessage

      • DeleteMessageBatch

      • ReceiveMessage

    • S3 Permissions

      • GetObject

    • Expel Permissions

      {
   “Version”: “2012-10-17”,
   “Statement”: [
   {
           “Action”: [
           “sqs:DeleteMessage”,
           “sqs:DeleteMessageBatch”,
           “sqs:ReceiveMessage”
           ],
           “Effect”: “Allow”,
           “Resource”: “YOUR_SQS_QUEUE_ARN”
    },
    {
           “Action”: [
           “s3:GetObject”
    ],
           “Effect”: “Allow”,
           “Resource”: “YOUR_S3_BUCKET_ARN/*”
    }
  ]
}

  3. Congratulations! You configured S3 notifications to an SQS queue through SNS. Contact your engagement manager for details on how to finish onboarding.

  • Was this article helpful?

  • 0 out of 1 found this helpful

Related articles