The Expel Amazon Elastic Kubernetes Service (EKS) consumes audit logs from the AWS platform through Kinesis. This visibility allows Workbench to identify activity of interest in EKS, investigate, and notify organizations if action is recommended.

Kub_Amazon_1_Overview.png

Resources

Review the following list of resources and their purposes:

Resource

Purpose

Kinesis Stream

MDR

Kinesis Stream Filter

MDR

IAM Role

MDR

IAM Policy “Puts” (binds)

Inventory/metrics

EKS ClusterRole

Security benchmarking

EKS ClusterRoleBinding

Security benchmarking

EKS IAM Identity Mapping

Security benchmarking

Network Allow Rule

All Workbench investigation access

Configuring MDR

When configuring MDR, use the following list to track your progress:

Enable control plane logging for each EKS cluster

At a minimum, Workbench requires logging of the audit events for each cluster. This provides visibility into activity affecting resources in the cluster: the “who, what, and when” we need to detect and take action.

Update logging configuration for each cluster in each region with the AWS CLI as in the following example:

CLUSTER_REGION=<your-cluster-region> ; \
CLUSTER_NAME=<your-cluster-name> ; \
aws eks update-cluster-config \
  --region ${CLUSTER_REGION} \
  --name ${CLUSTER_NAME} \
  --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'

For more information on configuring control plane logging, see the Amazon reference guide.

After control plane logging is enabled, EKS begins sending logs to CloudWatch.

Create a Kinesis Stream

Logs must be routed from CloudWatch to a Kinesis data stream so Workbench can consume the logs in real time.

Create a stream with the AWS CLI tool:

CLUSTER_REGION=<your-cluster-region> ; \
K_STREAM_NAME=expel-aws-eks-kinesis-data-stream ; \
aws kinesis create-stream \
  --region ${CLUSTER_REGION} \
  --stream-name ${K_STREAM_NAME} \
  --stream-mode-details '{"StreamMode": "ON_DEMAND"}'

Note

The value of K_STREAM_NAME must be different for each separate region.

Tip

Note the Kinesis Stream ARN that the code CLI generates for use in later steps. The ARN is different for reach region.

CLUSTER_REGION=<your-cluster-region> ; \
K_STREAM_NAME=expel-aws-eks-kinesis-data-stream ; \
aws kinesis describe-stream-summary --stream-name=${K_STREAM_NAME} --region=${CLUSTER_REGION}

Tip

We recommend creating an “ON_DEMAND” stream to allow the stream to adjust capacity based on demand.

Create an IAM role for CloudWatch log delivery

An IAM role is required to allow CloudWatch to deliver logs to your Kinesis stream.

  1. Create a trust policy document that allows the CloudWatch service to assume the role.

    The following sample code writes a policy JSON definition to expel_cw_assume_role_trust
    _policy_doc.json
    in the current directory.

    Note

    For convenience, the system creates a JSON document named expel_cw_assume_role_trust_policy_doc.json on the local machine.

    AWS_REGION=<your-aws-region> ; \
    AWS_ACCOUNT_ID=<your-aws-account-id> ; \
    cat > expel_cw_assume_role_trust_policy_doc.json << EOF
    {
      "Statement": {
        "Effect": "Allow",
        "Principal": { "Service": "logs.${AWS_REGION}.amazonaws.com" },
        "Action": "sts:AssumeRole",
        "Condition": {
          "StringLike": { "aws:SourceArn": "arn:aws:logs:${AWS_REGION}:${AWS_ACCOUNT_ID}:*" }
        }
      }
    }
    
    EOF
  2. Create the IAM role with the expel_cw_assume_role_trust_policy_doc.json document:

    EXPEL_CW_ASSUME_ROLE_NAME=expel-aws-eks-cloudwatch-assume-role ; \
    aws iam create-role \
      --role-name ${EXPEL_CW_ASSUME_ROLE_NAME} \
      --assume-role-policy-document file://expel_cw_assume_role_trust_policy_doc.json 

Create an IAM policy document for the producer role

Create an IAM policy document that grants the following role permissions to put records into your Kinesis stream:

K_STREAM_ARN=<kinesis stream ARN saved in earlier step> ; \
cat > expel_iam_policy_doc.json << EOF
{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "kinesis:PutRecord",
      "Resource": "${K_STREAM_ARN}"
    }
  ]
}

EOF

Apply policy to producer role

Apply the IAM policy to the CloudWatch role:

EXPEL_CLOUDWATCH_ROLE_NAME=expel-aws-eks-cloudwatch-assume-role; \
EXPEL_IAM_POLICY_NAME=expel-aws-eks-producer-policy; \
aws iam put-role-policy \
   --role-name "${EXPEL_CLOUDWATCH_ROLE_NAME}" \
   --policy-name "${EXPEL_IAM_POLICY_NAME}" \
   --policy-document file://expel_iam_policy_doc.json

Route logs from CloudWatch to Kinesis

Logs must be routed to Kinesis with a CloudWatch log group subscription filter.

Create the filter with the AWS CLI:

K_STREAM_ARN=<kinesis stream ARN saved in earlier step> ; \
EKS_LOG_GROUP_NAME=<your choice of log group name> ; \
CLUSTER_AWS_REGION=<your-aws-region> ; \
EXPEL_CLOUDWATCH_ROLE_NAME=expel-aws-eks-cloudwatch-assume-role ; \
aws logs put-subscription-filter \
  --region ${CLUSTER_AWS_REGION} \
  --log-group-name ${EKS_LOG_GROUP_NAME} \
  --filter-name "AllEKSLogs" \
  --filter-pattern "" \
  --destination-arn ${K_STREAM_ARN} \
  --role-arn ${EXPEL_CLOUDWATCH_ROLE_NAME}

Create an IAM Role for Workbench

To authenticate to your AWS account and retrieve logs, Workbench requires an IAM role. If you already have an Workbench IAM role for another Workbench integration (like CloudWatch or Amazon GuardDuty), this existing role can be reused and you can go to Grant Workbench IAM role permissions.

  1. Create a trust policy document that allows Workbench to assume the role:

    Note

    For convenience, the system creates a JSON document named expel_wb_iam_role_policy_doc.json on the local machine.

    Tip

    You can find the organization GUID on the My Organizations tab.

    ORG_GUID=<Your workbench org_guid(formerly customer id)> ; \
    cat > expel_wb_iam_role_policy_doc.json << EOF
    {
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": { "AWS": "arn:aws:iam::012205512454:user/ExpelCloudService" },
          "Action": "sts:AssumeRole",
          "Condition": {
            "StringEquals": { "sts:ExternalId": "${ORG_GUID}" }
          }
        }
      ]
    }
    
    EOF
  2. Create the IAM role with the expel_wb_iam_role_policy_doc.json document:

    EXPEL_ASSUME_ROLE_NAME=ExpelServiceAssumeRole ; \
          aws iam create-role \
            --role-name ${EXPEL_ASSUME_ROLE_NAME} \
            --assume-role-policy-document file://expel_wb_iam_role_policy_doc.json

Grant Workbench IAM role permissions

Workbench requires IAM permissions to retrieve logs from Kinesis and to investigate the activity of interest affecting your EKS clusters.

  1. Create an IAM policy document that grants Workbench the required permissions for Kinesis and EKS:

    K_STREAM_ARN=< kinesis stream ARN saved in earlier step > ; \
    cat > expel_k8s_ro_policy_doc.json << EOF
    {
        "Version": "2012-10-17",
        "Statement": [
          {
                "Effect": "Allow",
                "Action": [
                    "kinesis:DescribeLimits",
                    "kinesis:DescribeStream",
                    "kinesis:DescribeStreamSummary",
                    "kinesis:GetRecords",
                    "kinesis:GetShardIterator",
                    "kinesis:ListShards"
                ],
              "Resource": "${K_STREAM_ARN}"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "eks:AccessKubernetesApi",
                    "eks:DescribeCluster",
                    "eks:DescribeNodegroup",
                    "eks:ListClusters",
                    "eks:ListNodegroups",
                    "eks:ListUpdates",
                    "sts:GetCallerIdentity",
                    "ec2:DescribeRegions",
                    "autoscaling:DescribeAutoScalingGroups"
                ],
                "Resource": "*"
            }
        ]
    }
    
    EOF
  2. Apply the policy to the Expel IAM role:

    EXPEL_ASSUME_ROLE_NAME=ExpelServiceAssumeRole ; \
    EXPEL_EKS_CONSUMER_POLICY_NAME=expel-aws-eks-eks-consumer-policy ; \
    aws iam put-role-policy \
      --role-name ${EXPEL_ASSUME_ROLE_NAME} \
      --policy-name ${EXPEL_EKS_CONSUMER_POLICY_NAME} \
      --policy-document file://expel_k8s_ro_policy_doc.json

Enabling Expel security benchmark report

When enabling security benchmark reports, use the following list to track your progress:

Grant Workbench read-only cluster permissions

Expel suggests providing limited read-only access to your EKS clusters to enable deep investigation of interesting activity. These permissions can be fine-tuned based on your needs. Workbench can deliver service without this access, but it significantly limits our ability to thoroughly investigate activity in your clusters.

Create ClusterRole and ClusterRoleBinding

An Expel ClusterRole manifest grants limited, read-only permissions.

  1. Create an Expel ClusterRole manifest by saving the following code as expel-cluster-role.yml:

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: expel-reader-clusterrole
    rules:
    - apiGroups:
      - ""
      - admissionregistration.k8s.io
      - apps
      - networking.k8s.io
      - rbac.authorization.k8s.io
      resources:
      - apiservices
      - clusterrolebindings
      - clusterroles
      - cronjobs
      - daemonsets
      - deployments
      - events
      - flowschemas
      - horizontalpodautoscalers
      - ingressclasses
      - ingresses
      - jobs
      - localsubjectaccessreviews
      - mutatingwebhookconfigurations
      - namespaces
      - networkpolicies
      - nodes
      - persistentvolumes
      - poddisruptionbudgets
      - pods
      - podsecuritypolicies
      - podtemplates
      - replicasets
      - rolebindings
      - roles
      - selfsubjectaccessreviews
      - selfsubjectrulesreviews
      - serviceaccounts
      - services
      - statefulsets
      - subjectaccessreviews
      - tokenreviews
      - validatingwebhookconfigurations
      - volumeattachments
      verbs:
      - get
      - list
  2. Use kubectl to apply the role:

    kubectl apply -f expel-cluster-role.yml
  3. Create a cluster role binding by saving the following code as expel-cluster-role-binding.yml:

    kind: ClusterRoleBinding
    apiVersion: rbac.authorization.k8s.io/v1
    metadata:
      name: expel-reader-clusterrolebinding
    subjects:
    - kind: User
      name: expel-user
    roleRef:
      kind: ClusterRole
      name: expel-reader-clusterrole
      apiGroup: rbac.authorization.k8s.io
  4. Use kubectl to apply the role binding:

    kubectl apply -f expel-cluster-role-binding.yml

Map the Expel Role ARN to a cluster user using eksctl:

CLUSTER_REGION=<your-cluster-region> ; \
CLUSTER_NAME=<your-cluster-name> ; \
EXPEL_SERVICE_ASSUME_ROLE_ARN=<expel-service-assume-role-arn> ; \
eksctl create iamidentitymapping \
  --cluster ${CLUSTER_NAME} \
  --region ${CLUSTER_REGION} \
  --arn ${EXPEL_SERVICE_ASSUME_ROLE_ARN} \
  --username expel-user

Confirm the mapping with the following code:

CLUSTER_REGION=<your-cluster-region> ; \
CLUSTER_NAME=<your-cluster-name> ; \
eksctl get iamidentitymapping \
  --cluster ${CLUSTER_NAME} \
  --region ${CLUSTER_REGION}

Configuring Workbench

When connecting to Workbench, use the following checklist to track your progress:

Add a security device to Workbench

  1. Log in to https://workbench.expel.io.

  2. Navigate to Organization Settings > Security devices.

  3. Click +Add security device.

  4. Select Amazon Elastic Kubernetes Service (EKS).

    Kub_Amazon_2_AddSecDev.png
  5. Complete the Add Security Device form:

    • Name: the name of the device.

    • Location: the location of the device.

    • Role ARN: the ARN for the cluster role you created in Create an IAM role for Workbench.

    • Role Session Name: we recommend ExpelSession.

    • Region: the AWS region with the Kubernetes cluster.

    • Kinesis Stream Name: the log stream name you created in Create a Kinesis Stream.

Enable network access

Put the following IP addresses in your allow list:

Tip

This page was accurate at the time of writing, but changes happen. If you find the instructions are outdated, let us know via your engagement manager or account representative.