Note
This manual setup guide is for those who do not have an existing trail available for use in their CloudTrail service. The AWS setup described here is Expel’s recommended best practice.

 

This onboarding guide takes you through all required manual configurations when creating a new trail in AWS, to allow Expel to access your S3 bucket securely via an API. It also shows you how to add CloudTrail as a security device in Workbench, which completes the integration and enables you to monitor all activity.

If you would like to use the Wizard instead, see Setup Options

Prerequisites

  1. Verify you have the necessary permissions to create and modify IAM policies and roles for your AWS account.
  2. Make sure you are in the management console if you are using AWS Organizations, or are in the proper AWS account if you manage more than one account.
  3. Check the AWS region in the top menu bar (i.e. us-east-2, us-west-1, etc.) and make sure you are in the home region you want to be in; you must use the same home region throughout the entire AWS configuration process, and you must later specify this region correctly in Workbench.
  4. Make sure you can log into Workbench successfully, and are able to see your organization settings.

    Note
    This is a good time to copy and save your Workbench GUID, which is a unique alphanumeric value assigned by Expel to your organization and found in the My Organization page; learn more about organizations.
  5. Create a new file or other space to keep track of all of the names, ARNs, and other values from the AWS configuration as you go (a list of all needed values is available in the Quick Start, and additional help is available in the Reference).

Quick Start

Note
There are a number of ways to customize your AWS configurations based on your own environment or protocols. These steps are intended to show the necessary settings that must be configured in order for your Expel integration to work, with basic instructions for how to do so; they do not cover most of the optional settings or explain every possible option in AWS.

 

Setup includes the following steps (select any step for detailed instructions):

  1. Create a new trail with a new S3 bucket and new AWS KMS key.
  2. Edit the S3 bucket’s default encryption settings to use your AWS KMS key.
  3. Create a new SNS topic to receive events from the S3 bucket.
  4. Edit the AWS KMS Key policy to enable communication to the SNS topic.
  5. Create the SQS queue to receive messages from the SNS topic.
  6. Create the SNS subscription to send events to the SQS queue.
  7. Enable S3 bucket event notifications to send certain events to the SNS topic.
  8. Set up CloudFormation to configure the IAM role and policy.
  9. Create an additional stack to finish IAM role and policy configuration (AWS Organizations only).
  10. Add AWS CloudTrail as a security device in Workbench.

During the AWS configuration process, you will need to copy out and save a number of values as they are added by you (e.g. the AWS KMS Alias) or generated by AWS (e.g. the SQS Queue ARN). These include, in the order they are created or generated:

  • Workbench GUID (to save time, get it now)
  • AWS Region
  • S3 Bucket Name
  • AWS KMS Alias
  • S3 Bucket ARN
  • SNS Topic ARN
  • AWS KMS Key ARN
  • SQS Queue ARN
  • SQS URL
  • IAM Role ARN

Knowing each of these values is necessary to successfully complete all AWS configuration steps and to also add AWS as a security device in Workbench.


Note
For examples of what some of these values look like or instructions on where to find them if you forget to save one of them, refer to the Reference section.

Step 1: Create a New Trail

Before you begin, make sure you are in the proper AWS account (if you manage more than one AWS account) and that you are in the management console (if you use AWS Organizations). Also, check the top menu bar to be sure you are in the home region you wish to set for the trail (e.g. us-west-1). You will need to be in the same home region throughout this guide, and also choose the correct region in Workbench in Step 10.

In this step, you will create a new trail, new S3 bucket, and new AWS KMS key within your chosen home region.

  1. Use the Search bar to quickly navigate to Cloudtrail, or find it in the Services menu.
  2. Select Create trail.
  3. Enter a trail name. 
    • Suggestion: ExpelCloudTrail
  4. AWS Organizations only - select the Enable for all accounts in my organization checkbox.
  5. In Storage location, leave Create new s3 bucket selected.
    • Use the default Trail log bucket and folder, or enter a new unique name.

      Note
      Remember that if you choose to create your own name, it must be unique to ALL of AWS — not just to your instance — or the trail creation will fail. You can append your desired name with the alphanumeric values from the default name to ensure uniqueness.
    • Make a note of your S3 bucket name, as you will need it for the next section.
    • Leave Log file SSE-KMS encryption as Enabled.
    • Leave the New radio button selected.
    • Enter a name for AWS KMS alias.
    • Make a note of your AWS KMS alias, as you will need it in later steps.
  6. In Additional settings:
    • Leave Log file validation as Enabled.
    • Leave SNS notification delivery unchecked.
  7. Leave the CloudWatch Logs and Tags section as is, unless you need to change them for your own protocols, and do not edit the Policy document.
  8. Select Next.
  9. Leave the Choose Log Events screen as is.
  10. Select Next.
  11. You can now review your trail’s settings before creating it, if you wish.
  12. Select Create trail.
  13. You will be taken to the Trails list and should see your new trail.

Step 2: Edit the S3 Bucket Encryption Settings

In this step, you will set your default S3 bucket encryption to server-side encryption with AWS Key Management Service keys (SSE-KMS), and instruct Amazon to use your AWS KMS key’s encryption policy for your log files instead of using SSE-S3 encryption (Amazon’s default). You will configure the key’s policy in a later section.

  1. Use the Search bar to quickly navigate to S3, or find it in the Services menu.
  2. Look for your new S3 bucket in the list and select it.
  3. Select the Properties tab.
  4. First, copy the ARN and save it to a safe place as your S3 Bucket ARN (you will need it for the next section). The format should look something like this: arn:aws:s3:::YourS3BucketName
  5. Next, scroll to Default encryption and select Edit.
    • Select Server-side encryption with AWS Key Management Service keys (SSE-KMS).
    • Under AWS KMS key, select Choose from your AWS KMS keys.
    • Select the AWS KMS key you created in Step 1 (the alias will display in small print below the key).
    • Leave the Bucket Key as Enable.
    • Select Save changes.

Step 3: Create a New SNS Topic

Now, you will create an SNS topic and edit its access policy to give the S3 bucket permission to push events to the topic. These communications will eventually be sent from the SNS topic to an SQS queue (configured in a later section).

  1. Use the Search bar to quickly navigate to Simple Notification Service (SNS), or find it in the Services menu.
  2. Select Topics.
  3. Select Create topic.
  4. In the Details section:
    • Select Standard as the Type.
    • Enter a topic name in the Name field.
  5. Expand the Encryption section.
    • Toggle Encryption to on.
    • Use the dropdown menu to select your AWS KMS Key (or search for the alias if you do not see it).
  6. Expand the Access Policy section.
    • Leave the method as Basic.
    • Look at the JSON preview and scroll down to the Resource key. 
    • Copy the Resource value and save it to a safe place as your SNS Topic ARN (you will need it to configure the new access policy). The format should look something like this: arn:aws:sns:us-east-1:123456789012:YourTopicName
    • Now, select the Advanced button to define a new access policy.
    • Highlight and Delete the existing policy and paste the below policy instead. Make sure to use your SNS Topic ARN as the Resource value and your S3 Bucket ARN as the aws:SourceARN value.
{
  "Version": "2008-10-17",
  "Id": "expel-topic-policy-ID",
  "Statement": [
    {
      "Sid": "expel-statement-ID",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "SNS:Publish"
      ],
      "Resource": "YOUR_SNS_TOPIC_ARN",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "YOUR_S3_BUCKET_ARN"
        }
      }
    }
  ]
}

Note
Did you use your SNS Topic ARN and S3 bucket ARN in the new code? Check to be sure before continuing.

  1. Leave the remaining sections as is and select Create topic.

Step 4: Edit the AWS KMS Key Policy

In Step 1, you configured your S3 bucket to use your AWS KMS key for encryption. Now, you must append the key’s encryption policy to allow the S3 bucket and SNS topic to communicate securely via encryption.

  1. Use the Search bar to quickly navigate to Key Management Service, or find it in the Services menu.
  2. From the Customer Managed Keys list, select your AWS KMS alias.
  3. First, copy the ARN and save it to a safe place as your AWS KMS Key ARN (you will need it in a later step). The format should look something like this: arn:aws:kms:us-west-2:123456789012:key/123a4567-890b-1234-c5d6-7ef89012g345
  4. In the Key policy section, select Edit and append these JSON objects to the code. Make sure to use your S3 Bucket ARN and SNS Topic ARN as the aws:SourceARN values.
,
{ 
 "Sid": "Allow cloudtrail bucket to encrypt/decrypt",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
       "Action": [
               "kms:GenerateDataKey",
               "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "aws:SourceArn": "YOUR_S3_BUCKET_ARN"
                }
            }
        },
{
    "Sid": "Allow SNS to encrypt/decrypt",
    "Effect": "Allow",
    "Principal": {
        "Service": "sns.amazonaws.com"
    },
    "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
    "Resource": "*",
    "Condition": {
        "StringLike": {
            "aws:SourceArn": "YOUR_SNS_TOPIC_ARN"
        }
    }
}

If you need help appending your policy, go to the Troubleshooting section

 

Note
Did you use your S3 bucket ARN and SNS Topic ARN in the appended code? Check to be sure before continuing.

  1. Select Save changes.

Step 5: Create the SQS Queue

Next, we'll create a dedicated SQS queue to receive messages from the SNS topic, and edit the access policy to allow the SNS topic to send messages to it. Expel will use this queue to poll for notifications of new CloudTrail data, and then will update Workbench accordingly.

  1. Use the Search bar to quickly navigate to Simple Queue Service (SQS), or find it in the Services menu. 
  2. Select Create queue.
  3. In the Details section:
    • Leave Standard as the type.
    • Enter a name in the Name field.
  4. In the Configuration section:
    • Change the Message Retention period to 7 days
    • Leave all other defaults.
  5. In the Encryption section:
    • Leave Server-side encryption as Enabled.
    • Choose AWS Key Management Service key (SSE-KMS) as the encryption key type. 
    • Choose your AWS KMS Alias from the Customer master key dropdown. 
    • Leave the data key reuse period as the default.
  6. In the Access policy section:
    • Select Advanced.
    • First, go to the Resource value and copy the ARN, then save it to a safe place as your SQS Queue ARN (you will need it for your new access policy).
    • Highlight and delete the existing policy and paste the below policy instead. Make sure to use your SQS Queue ARN as the Resource value and your SNS Topic ARN as the aws:SourceARN value.
{
  "Version": "2012-10-17",
  "Id": "SQSDefaultPolicy",
  "Statement": [
    {
      "Sid": "Sid1572965666162",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SQS:SendMessage",
      "Resource": "YOUR_SQS_QUEUE_ARN",
      "Condition": {
        "StringEquals": {
          "aws:SourceArn": "YOUR_SNS_TOPIC_ARN"
        }
      }
    }
  ]
}

Note
Did you use your SQS Queue ARN and SNS Topic ARN in the new code? Check to be sure before continuing.

  1. Leave the remaining sections as is and select Create queue.
  2. Before leaving the confirmation screen, copy the URL and save it to a safe place as your SQS URL (you will need it in a later section).

Step 6: Create the SNS Subscription

Now that we have the SNS topic and SQS queue, and have also edited both of their access policies to allow for communication, we need to create a subscription within the SNS topic so that the topic can send events to the queue.

  1. Use the Search bar to quickly navigate to Simple Notification Service (SNS), or find it in the Services menu. 
  2. Select Subscriptions.
  3. Select Create subscription.
  4. In the Details section:
    • Select your SNS Topic ARN.
    • Select Amazon SQS as the protocol.
    • Select your SQS Queue ARN as the endpoint.
    • Select Enable raw message delivery checkbox.

      Note
      Selecting this option ensures SNS does not add metadata to the events it sends to SQS, so be sure this is enabled.
  5. Leave the remaining sections as is and select Create subscription.

Step 7: Enable S3 Event Notifications

Now that we've fully configured the communications between the SNS topic and SQS queue, we can create an event notification. This notification will tell the S3 bucket to send certain events (we will specify which ones) to the SNS topic whenever CloudTrail adds event logs to the bucket. 

  1. Use the Search bar to quickly navigate to S3, or find it in the Services menu.
  2. Select your S3 bucket.
  3. Select Properties.
  4. Go to the Event Notifications section and select Create event notification. Then:
    • Enter an event name.
    • In the Event types section, use the checkbox to enable All object create events. Leave the rest of the boxes unchecked.
    • In the Destination section, select SNS topic and then choose your SNS topic from the dropdown.
  5. Select Save changes.

Step 8: Set Up CloudFormation

Expel will authenticate its credentials using an IAM Role, and will be granted permissions to your AWS account(s) based on a corresponding IAM policy. You can complete this step most efficiently by using CloudFormation instead of doing a manual configuration via IAM. 

AWS Organizations

If you use AWS Organizations, you will create a stackset in the management account (this is where the primary Expel role resides). Creating a stackset allows AWS to automatically replicate and deploy the IAM role and policy via stacks to all of the other accounts in your organization. This enables Expel to perform all necessary investigative actions, and also prevents you from having to update or redo the configuration if a new account is added to your organization.

AWS Single Accounts

If you only have one AWS account, you will create a stack instead of a stackset. This is the recommended best practice because it is a faster configuration process that is less likely to fail due to human error. However, you may use IAM to create your policy and role manually if you wish, then go to Step 10 to continue the process.

AWS Organizations

These instructions are for AWS Organizations to create a stackset.

 

Note
If you have a single AWS account, scroll down to find instructions for creating your stack.

 

Note
Before you begin, make sure you have your Workbench GUID. If you need additional help finding this value, see the Reference for detailed instructions.

  1. Download the StackSet template, then come back to this topic when you have finished.
  2. Use the Search bar to quickly navigate to CloudFormation, or find it in the Services menu.
  3. Go to StackSets.
  4. Select Create StackSet.
    • If you see a dropdown menu, choose the With new resources option.
  5. For the template:
    • Leave Permissions as Service-managed permissions.
    • Leave the Prerequisite as Template is ready.
    • For the template source, choose Upload a template file.
    • Choose the stackset.json template file (or whatever you have named it, if you chose a different name).
  6. Select Next.
  7. Configure your stackset details.
    • Enter a name for your stackset.
    • Enter your Workbench GUID.
  8. Select Next.
  9. Leave Tags and Execution configuration as is.
  10. Select Next.
  11. For your deployment options:
    • Scroll down to Specify regions and choose your AWS region.
    • Leave all other defaults on this page as is.
  12. Select Next.
  13. Review your configuration if desired, and select Submit.

You can now go to Step 9.

AWS Single Accounts

These instructions are for AWS single account holders to create a stack. If you have an AWS organization and have already created your stackset, go to step 9.

Note
Before you begin, make sure you have your SQS Queue ARN, S3 Bucket ARN, AWS KMS Key ARN, and Workbench GUID. If you need additional help finding any of these values, see the Reference for detailed instructions.

  1. Download the Stack template, then come back to this topic when you have finished.
  2. Use the Search bar to quickly navigate to CloudFormation, or find it in the Services menu.
  3. Go to Stacks.
  4. Select Create StackSet.
    • If you see a dropdown menu, choose the With new resources option.
  5. For your stack:
    • Leave the Prerequisite as Template is ready.
    • For the template source, choose Upload a template file.
    • Choose the stack.json template file (or whatever you have named it, if you chose a different name).
  6. Select Next.
  7. Configure your stack details.
    • Enter a name for your stack.
    • Enter your AWS KMS Key ARN, S3 Bucket ARN, SQS Queue ARN, and Workbench GUID.
  8. Select Next.
  9. Leave all defaults for stack options as is.
  10. Select Next.
  11. Review your configuration if desired, and select Submit.
  12. Now, you must grab your IAM Role ARN (you will need it for the next section). Use the Search bar to quickly navigate to IAM, or find it in the Services menu.
  13. Go to Roles.
  14. Look for ExpelAssumeRole and select it.
  15. In the Summary section, copy the ARN and save it to a safe place as your IAM Role ARN.

You can now skip to Step 10.

Step 9: Create an Additional Stack (AWS Organizations only)

Note
If you have a single AWS account, you do not need this additional stack because you have already included all necessary permissions in the policy during Step 8. Skip to Step 10.

 

AWS Organizations will need to create an additional stack that is just for the management organization. This stack will enable additional permissions related to the SQS Queue ARN, S3 Bucket ARN, and KMS Key ARN.


Note
Before you begin, make sure you have your SQS Queue ARN, S3 Bucket ARN, AWS KMS Key ARN, and Workbench GUID. If you need additional help finding any of these values, see the Reference for detailed instructions.

  1. Download the Stack template, then come back to this topic when you have finished.
  2. Use the Search bar to quickly navigate to CloudFormation, or find it in the Services menu.
  3. Go to Stacks.
  4. Select Create StackSet.
    • If you see a dropdown menu, choose the With new resources option.
  5. For your stack:
    • Leave the Prerequisite as Template is ready.
    • For the template source, choose Upload a template file.
    • Choose the stack.json template file (or whatever you have named it, if you chose a different name).
  6. Select Next.
  7. Configure your stack details.
    • Enter a name for your stack.
    • Enter your AWS KMS Key ARN, S3 Bucket ARN, SQS Queue ARN, and Workbench GUID.
  8. Select Next.
  9. Leave all defaults for stack options as is.
  10. Select Next.
  11. Review your configuration if desired, and select Submit.
  12. Now, you must grab your IAM Role ARN (you will need it for the next section). Use the Search bar to quickly navigate to IAM, or find it in the Services menu.
  13. Go to Roles.
  14. Look for ExpelAssumeRole and select it.
  15. In the Summary section, copy the ARN and save it to a safe place as your IAM Role ARN.

Step 10: Add AWS CloudTrail as a Security Device in Workbench

Now, we can add a security device in Workbench to complete the integration.

 

Note
Before you begin, make sure you have your IAM Role ARN and SQS URL. If you use AWS Organizations, you will also need your AWS account number. If you need additional help finding any of these values, see the Reference for detailed instructions.

  1. Log in to Workbench.
  2. In the side menu, navigate to Organization Settings > Security Devices.
  3. Select Add Security Device.
  4. Select AWS Cloudtrail (or search for it if you do not see it listed). Then select the bullets as follows:
    • “Are you using AWS organizations?” - AWS Organizations should leave it as Yes; single accounts should select No.
    • “Do you have an existing CloudTrail that you want Expel to reuse?” - select No.
    • “How would you like to connect?” - select Manual connection.
    • Select Save.
  5. In the next screen, complete the fields as follows:
    • Enter a Name that might help you more easily identify this integration, such as “CompanyName CloudTrail”

      Note
      This name will display in Workbench under the Name column, and is a text string that you can filter on.
    • Enter the Location of your integration, for example “cloud” or “AWS cloud” or “on prem"

      Note
      This is also a text string that you can filter on, so we recommend being consistent with location naming across your Expel integrations.
    • Enter the IAM Role ARN you saved from the last section. Format: arn:aws:iam::123456789012:role/RoleName
    • Enter a text string to label your IAM role session. Example: orgname-expel-trail-session
    • Choose the AWS region where you created your configuration in AWS.
    • Enter the SQS URL you copied and saved in a previous step.
    • If you use AWS Organizations, input your AWS account number; single account users can leave this field blank.
    • Select Save.
  6. Your device should be created successfully within a few seconds. A few reminders:
    • After your connection is healthy, it will take some time for your device to begin polling and receiving data.
    • To check on the status, select the downward arrow for your device in the first column and choose View details. You can then scroll to the Connection section to see if your device is fully connected.
    • Polling will happen first; data will be received after that.

      Note
      You must refresh the page to see updates. 
    • If your device does not begin polling within 15 minutes, and does not begin receiving data within 30 minutes, contact our support team for help.

Troubleshooting

S3 Event Notification Errors (Step 5)

This error is encountered when selecting Save changes in Step 7. It generally displays for one of the following reasons:

  • You left one of your ARN values out of a JSON object in a prior step, put an ARN in the wrong location, or formatted an ARN incorrectly.
  • The JSON objects were copied, appended, or formatted incorrectly in a prior step.
  • You used an S3 bucket name that someone else in the AWS ecosystem is already using (even if nobody at your organization is using it).
  • The integration is misconfigured; review all steps to verify that the necessary settings are enabled and configured properly.
    • Examples: Not choosing your AWS KMS key from the dropdown menu when setting up default encryption in step 2, or not using the same region throughout your AWS configuration.

S3 Bucket Naming

You must use an S3 bucket name that nobody else in the AWS ecosystem is using. You can avoid this problem by using the default name or by appending your chosen name with the numerical values from the default name that is generated.

Appending JSON Files

Make sure to put this JSON snippet in the correct place, using a comma, and make sure the final bracket is still there. Lines 78-133 of your JSON code should look like this (the appended part is in orange):

  {
            "Sid": "Enable cross account log decryption",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "kms:Decrypt",
                "kms:ReEncryptFrom"
            ],
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "kms:CallerAccount": "484330702365"
                },
                "StringLike": {
                    "kms:EncryptionContext:aws:cloudtrail:arn": "arn:aws:cloudtrail:*:484330702365:trail/*"
                }
            }
        },
        {
            "Sid": "Allow cloudtrail bucket to encrypt/decrypt",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "aws:SourceArn": "arn:aws:s3:::S3BucketName"
                }
            }
        },
        {
            "Sid": "Allow SNS to encrypt/decrypt",
            "Effect": "Allow",
            "Principal": {
                "Service": "sns.amazonaws.com"
            },
            "Action": [
                "kms:GenerateDataKey",
                "kms:Decrypt"
            ],
            "Resource": "*",
            "Condition": {
                "StringLike": {
                    "aws:SourceArn": "arn:aws:sns:us-east-1:123456789012:SNSTopicName"
                }
            }
        }
    ]

Reference

ARNs and URLs

This chart gives examples of ARNs and the SQS URL so you can check on the general formatting, and also tells you where to find any of the values if you’ve forgotten to copy and save them during the configuration process. 

 

Value Where to Find Example
S3 Bucket ARN S3 > Buckets > bucket name > Properties arn:aws:s3:::MyS3BucketARN
SNS Topic ARN Simple Notification Service > Topics > topic name > Details section arn:aws:sns:us-east-1:123456789012:MyTopicName
AWS KMS Key ARN Key Management Service > key alias > General configuration section arn:aws:kms:us-west-2:123456789012:key/123a4567-890b-1234-c5d6-7ef89012g345
SQS Queue ARN Simple Queue Service > queue name > Details section arn:aws:sqs:us-east-1:123456789012:MySQSQueueName
SQS URL Simple Queue Service > queue name > Details section https://sqs.us-west-1.amazonaws.com/123456789012/MySQSQueueName
IAM Role ARN IAM > Access Management > Roles > role name > Summary section arn:aws:iam::123456789012:role/RoleName

(most users will look for the ExpelAssumeRole)

 

Workbench GUID

The Workbench GUID is a unique alphanumeric value assigned by Expel to your organization.

Format

a123b456-7c89-0def-g1hi-2j3k45l6mn7o

Where to Find

  1. Log in to Workbench.
  2. Go to Organization Settings > My Organization.
  3. On the organization's page, look for the Organization GUID and select the Copy button to copy the GUID.

Note
If you have multiple organizations, you must first select the organization name that will be associated with your CloudTrail integration to access the page with the Copy button. Or, you can stay on the page and highlight then copy the GUID shown for that organization in the GUID column.