1. Expel Help Center
  2. Connect Your Technology
  3. Cloud Integrations
  4. Amazon/AWS

AWS CloudTrail Manual Setup - Existing Trail

This manual setup guide is for those who have an existing trail available for use in their CloudTrail service. For AWS Organizations users, this guide assumes your SNS Topic, SQS Queue and KMS Key encrypting SNS are stored in the same account as your S3 bucket. It also assumes you have encrypted both your CloudTrail and S3 bucket with KMS (Key Management Service). Please see this guide's Reference section for more information.

This onboarding guide takes you through required manual configurations to allow Expel to access your S3 bucket securely via an API. It also shows you how to add CloudTrail as a security device in Workbench, which completes the integration and enables you to monitor all activity.

Prerequisites

  1. You must already have an existing trail that is configured to send CloudTrail events to an S3 bucket.
  2. Verify you have the necessary permissions to create and modify IAM policies and roles for your AWS account.
  3. Make sure you are in the management console if you are using AWS Organizations, or are in the proper AWS account if you manage more than one account.
  4. Check the AWS region in the top menu bar (i.e. us-east-2, us-west-1, etc.) and make sure you are in the home region you want to be in; you must use the same home region throughout the entire AWS configuration process, and you must later specify this region correctly in Workbench.
  5. Make sure you can log into Workbench successfully, and are able to see your organization settings.
    • Note: This is a good time to copy and save your Workbench GUID, which is a unique alphanumeric value assigned by Expel to your organization and found in the My Organization page; see this Reference topic for detailed instructions.
  6. Create a new file or other space to keep track of all of the names, ARNs, and other values from the AWS configuration as you go (a list of all needed values is available in the Quick Start, and additional help is available in this Reference section).

Quick Start

Note:

There are a number of ways to customize your AWS configurations based on your own environment or protocols. These steps are intended to show the necessary settings that must be configured in order for your Expel integration to work, with basic instructions for how to do so; they do not cover most of the optional settings or explain every possible option in AWS.

Setup includes the following steps (select any step for detailed instructions):

  1. Create a new KMS key
  2. Create a new SNS topic
  3. Edit the SNS SQS KMS Key Policy
  4. Create the SQS Queue
  5. Create the SNS Subscription
  6. Enable S3 Event Notifications
  7. Grant Expel IAM Role Access
  8. Add AWS CloudTrail as a Security Device in Workbench

During the AWS configuration process, you will need to reference and copy out and save a number of values. These include:

Values you should have already:

  • Workbench Unique ID (get it now or during Step 7)
  • AWS Region
  • S3 bucket ARN
  • S3 Bucket KMS Key ARN (KMS ARN of the key set as default encryption of your S3 bucket)
  • CloudTrail KMS Key ARN (KMS ARN of the key Cloudtrail uses to encrypt logs that are sent to S3 bucket)

Values you will acquire as you follow this guide:

  • SNS SQS KMS Key ARN
  • SNS Topic ARN
  • SQS Queue ARN
  • SQS URL
  • S3 bucket account IAM Role ARN

Knowing each of these values is necessary to successfully complete all AWS configuration steps and to also add AWS as a security device in Workbench. For help with finding AWS values, refer to this Reference section.

Step 1: Create a New KMS Key

First we will create a KMS key to use to encrypt our SNS Topic and SQS queue. Create this key in the same account as the S3 bucket account.

  1. Use the Search bar to quickly navigate to Key Management Service, or find it in the Services menu.
  2. Select Create key.
  3. Leave the Configure key page as is and select Next.
  4. Enter an alias or display name for the key.
  5. Select Next.
  6. Leave the Define key administrative permissions page as is and select Next.
  7. Leave the Define key usage permissions page as is and select Next.
  8. Review your configuration if desired.
  9. Select Finish.
  10. In the key list, select the key you just made. Copy the ARN and save it to a safe place as your SNS SQS KMS Key ARN.

 

Step 2: Create a New SNS Topic

Note:
If you have already Amazon Simple Notification Service (SNS) configured as an event notifier for your existing CloudTrail bucket that includes the event type “All object create events”, proceed to step 3.


Otherwise, you will create an SNS topic and edit its access policy to give the S3 bucket permission to push events to the topic. These communications will eventually be sent from the SNS topic to an SQS queue (configured in a later section).

  1. Use the Search bar to quickly navigate to Simple Notification Service (SNS), or find it in the Services menu.
  2. Select Topics.
  3. Select Create topic.
  4. In the Details section, select Standard as the Type.
  5. Enter a topic name in the Name field.
  6. Expand the Encryption section.
    • Toggle Encryption to on.
    • Use the dropdown menu to select the SNS SQS KMS key you just created in Step 1.
  7. Expand the Access policy section.
    1. Leave the method as Basic.
    2. Look at the JSON preview and scroll down to the Resource key.
    3. Copy the Resource value and save it to a safe place as your SNS Topic ARN (you will need it to configure the new access policy). The format should look something like this: arn:aws:sns:us-east-1:123456789012:YourTopicName
    4. Next, select Advanced to define a new access policy.
    5. Highlight and Delete the existing policy and paste the below policy instead. Make sure to use your SNS Topic ARN as the Resource value and your S3 bucket ARN as the aws:SourceARN value.

       
      {
  "Version": "2008-10-17",
  "Id": "expel-topic-policy-ID",
  "Statement": [
    {
      "Sid": "expel-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "SNS:Publish"
      ],
      "Resource": "YOUR_SNS_TOPIC_ARN",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "YOUR_S3_BUCKET_ARN"
        }
      }
    }
  ]
}
  8. Important: Did you use your SNS Topic ARN and S3 bucket ARN in the new code? Check to be sure before continuing.
  9. Leave the remaining sections as is and select Create topic.

Step 3: Edit the SNS SQS KMS Key Policy

Next, we need to edit the SNS SQS KMS key policy to include proper permissions for our SNS topic and S3 bucket.

  1. Use the Search bar to quickly navigate to Key Management Service, or find it in the Services menu.
  2. Select the key you created in Step 1.
  3. Under Key policy, select Switch to policy view.
  4. Select Edit.
  5. Append the following JSON snippet into the existing policy. Make sure to use your S3 bucket ARN and SNS Topic ARN where specified.
 ,
        {
            "Sid": "Allow cloudtrail bucket to encrypt/decrypt",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "YOUR_S3_BUCKET_ARN"
                }
            }
        },
        {
            "Sid": "Allow SNS to encrypt/decrypt",
            "Effect": "Allow",
            "Principal": {
                "Service": "sns.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "YOUR_SNS_TOPIC_ARN"
                }
            }
        }
  1. Important: Did you use your S3 bucket ARN and SNS Topic ARN in the code? Check to be sure before continuing.
  2. Select Save changes.

Step 4: Create the SQS Queue

Next, we'll create a dedicated SQS queue to receive messages from the SNS topic, and edit the access policy to allow the SNS topic to send messages to it. Expel will use this queue to poll for notifications of new CloudTrail data, and then will update Workbench accordingly.

  1. Use the Search bar to quickly navigate to Simple Queue Service (SQS), or find it in the Services menu.
  2. Select Queues.
  3. Select Create queue.
  4. In the Details section, select Standard as the type.
  5. Enter a name in the Name field.
  6. In the Configuration section:
    1. Change the Message Retention period to 7 days.
    2. Leave all other defaults.
  7. In the Encryption section:
    1. Leave Server-side encryption as Enabled.
    2. Choose AWS Key Management Service key (SSE-KMS) as the encryption key type.
    3. Choose the SNS SQS KMS key you created in Step 1.
    4. Leave the data key reuse period as is.
  8. In the Access policy section:
    1. Select Advanced.
    2. First, go to the Resource value and copy the ARN, then save it to a safe place as your SQS Queue ARN (you will need it for your new access policy).
    3. Highlight and delete the existing policy and paste the below policy instead. Make sure to use your SQS Queue ARN as the Resource value and your SNS Topic ARN as the aws:SourceARN value. 
    {
  "Version": "2012-10-17",
  "Id": "SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "Sid1572965666162",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "YOUR_SQS_QUEUE_ARN",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "YOUR_SNS_TOPIC_ARN"
        }
      }
    }
  ]
}
  9. Important: Did you use your SQS Queue ARN and SNS Topic ARN in the new code? Check to be sure before continuing.
  10. Leave the remaining sections as is and select Create Queue.
  11. Before leaving the confirmation screen, copy the URL and save it to a safe place as your SQS URL (you will need it in a later section).

Step 5: Create the SNS Subscription

Now that we have an SNS topic and SQS queue, and have also edited both of their access policies to allow for communication, we need to create a new subscription within the SNS topic so that the topic can send events to the queue.

  1. Use the Search bar to quickly navigate to Simple Notification Service (SNS), or find it in the Services menu.
  2. Select Subscriptions.
  3. Select Create subscription.
  4. In the Details section, complete the fields as follows:
    • Topic ARN - Select the Topic ARN you obtained in Step 2.
    • Protocol - Amazon SQS.
    • Endpoint - Select your SQS Queue ARN.
    • Select Enable raw message delivery. Selecting this option ensures SNS doesn’t add metadata to the notifications it sends to SQS, so be sure this is enabled.
  5. Leave the remaining sections as is and select Create subscription.

Step 6: Enable S3 Event Notifications

Note:
If you have already Amazon Simple Notification Service (SNS) configured as an event notifier for your existing CloudTrail bucket that includes the event type “All object create events”, proceed to step 7.

Now that we've fully configured the communications between the SNS topic and SQS queue, we can create an event notification. This notification will tell the S3 bucket to send certain events (we will specify which ones) to the SNS topic whenever CloudTrail adds event logs to the bucket.

  1. Use the Search bar to quickly navigate to S3, or find it in the Services menu.
  2. Select your S3 bucket.
  3. Select Properties.
  4. Go to the Event Notifications section and select Create event notification. Then:
    1. Enter a name.
    2. In the Event types section, use the checkbox to enable All object create events. Leave the rest of the boxes unchecked.
    3. In the Destination section, select SNS topic and choose your SNS topic from the dropdown.
  5. Select Save changes.

Step 7: Grant Expel IAM Role access

Expel will authenticate its credentials using an IAM Role and will be granted permissions to your AWS account(s) based on a corresponding IAM policy. You can complete this process most efficiently by using CloudFormation instead of doing a manual configuration via IAM.

The process you follow will depend on how your environment is configured. See below for a scenario that suits your environment. If you have questions or need additional guidance, please reach out to your Expel Engagement Manager.

 

I have a single AWS account.

Please follow step 8 of the AWS CloudTrail Manual Setup - New Trail instructions and then return to this guide.

 

I am using AWS Organizations and the Cloudtrail and S3 bucket are in the same account.

Please follow steps 8 and 9 of the AWS CloudTrail Manual Setup - New Trail instructions and then return to this guide.

 

I am using AWS Organizations and the CloudTrail and S3 Bucket are in different accounts.

This section assumes that your SNS Topic, SQS Queue and KMS Key encrypting SNS/SQS are stored in the same account as your S3 bucket, with your CloudTrail in the management account. Resources existing in additional accounts will require additional configuration not provided in this guide. Please see this guide's Reference section for additional information.

 

Note: Before you begin, make sure you have you have the following values:

    • SQS Queue ARN (from step 4)
    • S3 bucket ARN (pre-existing)
    • KMS ARN your CloudTrail uses to encrypt the S3 bucket logs (pre-existing)
    • Workbench GUID.
      • If you need additional help finding these values, see the Reference for detailed instructions.

1. Use CloudFormation to deploy a stack template to deploy generic IAM policies in your management account:

  1. Download the Stack template, then come back to this guide when you have finished.
  2. Log into your management account. Please also ensure you are still in the same region as in previous steps.
  3. Use the Search bar to quickly navigate to CloudFormation, or find it in the Services menu.
  4. Go to Stacks.
  5. Select Create stack. If you see a dropdown menu, choose the "With new resources" option.
  6. For your stack:
    • Leave the Prerequisite as Choose an existing template.
    • For the template source, choose Upload a template file.
    • Choose the stack.json template file (or whatever you have named it, if you chose a different name).
  7. Select Next.
  8. Configure your stack details.
    • Enter a name for your stack.
    • Enter your KMS Key ARN your CloudTrail uses to encrypt S3 bucket logs, S3 Bucket ARN, SQS Queue ARN, and Workbench GUID.
  9. Select Next.
  10. Leave all defaults for stack options as is.
  11. Select Next.
  12. Review your configuration if desired, and select Submit.

2. Still logged into the management account, use CloudFormation to deploy a StackSet template to create generic IAM policies in all children accounts, including the S3 bucket account:

  1. Download the StackSet template, then come back to this topic when you have finished.
  2. Use the Search bar to quickly navigate to CloudFormation, or find it in the Services menu.
  3. Go to StackSets.
  4. Select Create StackSet. If you see a dropdown menu, choose the "With new resources" option.
  5. For the template:
    • Leave Permissions as Service-managed permissions.
    • Leave the Prerequisite as Template is ready.
    • For the template source, choose Upload a template file.
    • Choose the stackset.json template file (or whatever you have named it, if you chose a different name).
  6. Select Next.
  7. Configure your stackset details.
    • Enter a name for your stackset.
    • Enter your Workbench GUID.
  8. Select Next.
  9. Leave Tags and Execution configuration as is.
  10. Select Next.
  11. For your deployment options:
    • Scroll down to Specify regions and choose your AWS region.
    • Leave all other defaults on this page as is.
  12. Select Next.
  13. Review your configuration if desired, and select Submit.

3. Add a new inline policy to a role in the S3 bucket account to allow permissions to other resources:

Note: Before you begin, make sure you have you have the following values:

    • SQS Queue ARN (from step 4)
    • S3 bucket ARN (pre-existing)
    • KMS ARN your CloudTrail uses to encrypt S3 bucket logs (pre-existing)
    • Workbench GUID.
  1. Log into the child account the S3 bucket is in.
  2. Use the Search bar to quickly navigate to IAM, or find it in the Services menu.
  3. Select Roles.
  4. In the Roles list, search for or select ExpelAssumeRole.
  5. In the Summary section, copy the ARN and save it to a safe place as your S3 bucket account IAM Role ARN. We will need this value later when we add the CloudTrail in Workbench.
  6. In the Permissions policies section, select Add permissions > Create inline policy.
  7. In the JSON editor section, select JSON.
  8. Highlight and Delete the existing policy and paste the below policy instead. Make sure to use your SQS ARN from step 4, S3 bucket ARN, the KMS ARN encrypting your S3 bucket, your SNS SQS KMS ARN from step 1, and the KMS ARN for the key encrypting your Cloudtrail.

     
    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "sqs:DeleteMessage",
        "sqs:ReceiveMessage"
      ],
      "Resource": "YOUR_SQS_QUEUE_ARN"
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "YOUR_S3_BUCKET_ARN/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": "YOUR_SNS_SQS_KMS_ARN"
    },
    {
      "Effect": "Allow",
      "Action": [
        "kms:Decrypt"
      ],
      "Resource": "YOUR_CLOUDTRAIL_KMS_KEY_ARN"
    }
  ]
}
  9. Important: Did you use your SQS ARN, S3 bucket ARN, the KMS ARN encrypting your S3 bucket, your SNS SQS KMS ARN you created in Step 1, and the ARN of the KMS key encrypting your CloudTrail in the new code? Check to be sure before continuing.
  10. Select Next.
  11. Name your policy.
  12. Select Create policy.

4. Update the KMS Key Policy of the key that your CloudTrail uses to encrypt S3 bucket logs to give the Expel IAM Role in the S3 bucket account permissions to decrypt:

  1. Switch to your management account.
  2. Use the Search bar to quickly navigate to Key Management Service, or find it in the Services menu.
  3. Select the KMS key encrypting your CloudTrail.
  4. In the Key policy section, select Edit.
  5. Append the following JSON snippet to the Key policy, being sure to insert the account ID of the child account your S3 bucket resides in where specified:

    ,
       {
           "Sid": "Allow Expel to use this KMS key to decrypt logs",
           "Effect": "Allow",
           "Principal": {
               "AWS": "arn:aws:iam::YOUR_S3_BUCKET_ACCOUNT_ID:role/ExpelAssumeRole"
           },
           "Action": "kms:Decrypt",
           "Resource": "*"
       }
  6. Important: Did you use the account ID for the account your S3 bucket resides in in the new code? Check to be sure before continuing.
  7. Select Save changes.

 

Step 8: Add AWS CloudTrail as a Security Device in Workbench

  1. Log in to Workbench.
  2. In the side menu, navigate to Organization Settings > Security Devices.
  3. Select Add Security Device.
  4. Select AWS Cloudtrail (or search for it if you do not see it listed). Then select the bullets as follows:
    1. “Are you using AWS organizations?” - AWS Organizations should leave it as Yes; single accounts should select No.
    2. “Do you have an existing CloudTrail that you want Expel to reuse?” - leave it as Yes.
    3. “How would you like to connect?” - select Manual connection.
    4. Select Save.
  5. In the next screen, complete the fields as follows:
    1. Name - enter a name that might help you more easily identify this integration, such as “CompanyName CloudTrail”; this name will display in Workbench under the Name column, and is a text string that you can filter on.
    2. Location - enter the location of your integration, for example “cloud” or “AWS cloud” or “on prem;” this is also a text string that you can filter on, so we recommend being consistent with location naming across your Expel integrations.
    3. Role ARN - enter the IAM Role ARN you saved from the last section. Format: arn:aws:iam::123456789012:role/RoleName
    4. Role session name - enter a text string to label your IAM role session. Example: orgname-expel-trail-session
    5. AWS region - choose the AWS region where you created your configuration in AWS.
    6. SQS URL - enter the SQS URL you copied and saved in a previous step.
    7. Organization management account - if you use AWS Organizations, input the account ID of the management account; single account users can leave this field blank.
    8. Select Save.
  6. Your device should be created successfully within a few seconds. A few reminders:
    • After your connection is healthy, it will take some time for your device to begin polling and receiving data.
    • To check on the status, select the downward arrow for your device in the first column and choose View details. You can then scroll to the Connection section to see if your device is fully connected.
    • Polling will happen first; data will be received after that. You must refresh the page to see updates.
    • If your device does not begin polling within 15 minutes, and does not begin receiving data within 30 minutes, contact our support team for help.

Troubleshooting

If you are encountering an error or your Trail isn't connecting to Workbench, try checking for some common issues:

  • You left one of your ARN values out of a JSON object in a prior step, put an ARN in the wrong location, or formatted an ARN incorrectly.
  • The JSON objects were copied, appended, or formatted incorrectly in a prior step.
  • You have forgotten to enable something, or have configured something incorrectly, in one of the prior steps.
    • Examples: Not choosing your AWS KMS key from the dropdown menu when setting up default encryption in step 2, or not using the same region throughout your AWS configuration.

Reference

Cross-account configuration resources
The diagram below illustrates the relationship and location of AWS resources for a possible environment covered in this guide: an existing CloudTrail configuration involving AWS Organizations where the trail is in the management account and the S3 bucket, SNS topic, and SQS queue are all together in a child account. KMS encryption can be applied as illustrated and IAM roles are deployed to the accounts using CloudFormation Stacks and Stacksets.

AWS Cloudtrail design schematic - Data flow & IAM (1).png

For more information on configuring a cross-account environment, refer to these AWS developer guide pages:

  • Was this article helpful?

  • 1 out of 3 found this helpful

Related articles