Amazon Elastic Kubernetes Service (EKS) setup for Workbench

The Expel Amazon Elastic Kubernetes Service (EKS) consumes audit logs from the AWS platform through Kinesis. This visibility allows Workbench to identify activity of interest in EKS, investigate, and notify organizations if action is recommended.

Complete the steps below for each EKS cluster in each AWS account to be monitored by Workbench.

To complete this procedure you need:

Enable control plane logging for each EKS cluster

At a minimum, Workbench requires logging of the audit events for each cluster. This provides visibility into activity affecting resources in the cluster (the “who, what, and when” we need to detect and respond to).

Update logging configuration with the AWS CLI like this:

aws eks update-cluster-config \
    --region <your-cluster-region> \
    --name <your-cluster-name> \ 
       --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'

For more information on configuring control plane logging, see the Amazon reference guide.

Send EKS logs to a Kinesis stream

After control plane logging is enabled, EKS begins sending logs to CloudWatch. Next, the logs must be routed from CloudWatch to a Kinesis data stream so that Workbench can consume the logs in real time. Complete the steps below to route EKS logs to a Kinesis data stream.

Step 1: Create a new Kinesis data stream

Create a stream with the AWS CLI like this:

aws kinesis create-stream \
    --region <your-cluster-region> \
    --stream-name <your-stream-name> \
    --stream-mode-details '{"StreamMode": "ON_DEMAND"}'

We recommend creating an “ON_DEMAND” stream to allow the stream to adjust capacity based on demand.

Step 2: Create an IAM role for CloudWatch log delivery

An IAM role is required to allow CloudWatch to deliver logs to your Kinesis stream.

To start, create a trust policy document that allows the CloudWatch service to assume the role:

      {
   
    
        "Statement": {
    

          "Effect": "Allow",
    
        "Principal": { "Service": "logs. <your-region> .amazonaws.com" },
    
          "Action": "sts:AssumeRole",
    
    
          "Condition": { 
    
            "StringLike": { "aws:SourceArn": "arn:aws:logs: <your-region> : <your-account-id> :*" } 
    
           }
    
    
         }
    
    
      }
    

Then, create the IAM role supplying this trust document:

    
      aws iam create-role \
    
    --role-name <your-cloudwatch-role-name>
        --assume-role-policy-document file:// <your-trust-document-file-path>

Next, create an IAM policy document that grants this role permissions to put records into your Kinesis stream:

    
      {
    
    
        "Statement": [
    
    
          {
    
    
            "Effect": "Allow",
    

            "Action": "kinesis:PutRecord",
    
          "Resource": " <your-kinesis-arn> "
    
          }
    
    
        ]
    
    
      }
    

Finally, apply this IAM policy to the CloudWatch role:

aws iam put-role-policy \
     --role-name <your-cloudwatch-role-name> \
     --policy-name “<your-policy-name>” \
    --policy-document file://<your-policy-document-path>

Step 3: Route logs from CloudWatch to Kinesis stream

Logs can be routed to Kinesis with a CloudWatch log group subscription filter. Create this filter with the AWS CLI like this:

aws logs put-subscription-filter \
    --region <your-cluster-region> \
    –log-group-name <your-cluster-log-group-name> \
     –filter-name “AllEKSLogs” \
     –filter-pattern “”
    –destination-arn <your-kinesis-stream-arn>
    --role-arn <your-cloudwatch-role>

Create IAM Role for Workbench

To authenticate to your AWS account and retrieve logs, Workbench requires an IAM role. If you already have an Workbench IAM role for another Workbench integration (like CloudWatch or AWS GuardDuty), this existing role can be reused and you can go to: Grant Expel IAM Role required permissions.

To start, create a trust policy document that allows Workbench to assume the role:

      
      {
    
      
          "Statement":
    
      
          [
    
    
              {
    
    
                  "Effect": "Allow",
    
    
                  "Principal":
    
    
                  {
    
    
                      "AWS": "arn:aws:iam::012205512454:user/ExpelCloudService"
    
    
                  },
    
    
                  "Action": "sts:AssumeRole",
    
    
                  "Condition":
    
    
                  {
    
    
                      "StringEquals":
    
    
                      {
    
                        "sts:ExternalId": " <your-customer-guid> "
    
                      }
    
    
                  }
    
    
              }
    
    
          ]
    
    
      }
    

Then, create the IAM role supplying this trust document:

      aws iam create-role \
    
        --role-name <your-expel-role-name>
        --assume-role-policy-document file:// <your-trust-document-file-path>

Grant Workbench IAM Role required permissions

Workbench requires IAM permissions to retrieve logs from Kinesis, as well as investigate activity of interest affecting your EKS clusters.

First, create an IAM policy document granting Workbench required permissions for Kinesis and EKS:

    
      {
    
    
          "Version": "2012-10-17",
    
    
          "Statement": [
    
            {
    
                  "Effect": "Allow",
    
    
                  "Action": [
    
    
                      "kinesis:DescribeLimits",
     
    
                      "kinesis:DescribeStream",
    
    
                      "kinesis:DescribeStreamSummary",
    
    
                      "kinesis:GetRecords",
    
    
                      "kinesis:GetShardIterator",
    
    
                      "kinesis:ListShards"
    
    
                  ],
    
                "Resource": " <your kinesis stream arn> "
     
              },
    
    
              {
    
     
                  "Effect": "Allow",
    
    
                  "Action": [
    
    
                      "eks:AccessKubernetesApi",
    
    
                      "eks:DescribeCluster",
    
    
                      "eks:DescribeNodegroup",
    
    
                      "eks:ListClusters",
     
    
                      "eks:ListNodegroups",
    
    
                      "eks:ListUpdates",
    
    
                      "sts:GetCallerIdentity",
    
    
                      "ec2:DescribeRegions",
    
    
                      "autoscaling:DescribeAutoScalingGroups"
     
                  ],
    
    
                  "Resource": "*"
    
    
              }
    
    
          ]
    
    
      }
    

Then, apply this policy to the Expel IAM role:

aws iam put-role-policy \
          --role-name <your-expel-role-name> \ 
        --policy-name “<your-policy-name>” \
        --policy-document file://<your-policy-document-path>

Grant Workbench read-only cluster permissions

Expel suggests providing limited read-only access to your EKS clusters to enable deep investigation of interesting activity. These permissions can be fine-tuned based on your needs. Workbench can deliver service without this access, but it significantly limits our ability to thoroughly investigate activity in your clusters.

First, create an Expel ClusterRole manifest to grant limited, read-only permissions:

    
      apiVersion: rbac.authorization.k8s.io/v1
    
    
      kind: ClusterRole
    
    
      metadata:
    
    
        name: expel-reader-clusterrole
        
      rules:
    
    
      - apiGroups:
    
    
        - ""
    
    
        - admissionregistration.k8s.io
    
    
        - apps
    
    
        - networking.k8s.io
    
    
        - rbac.authorization.k8s.io
    
    
        resources:
    
    
        - apiservices
    
    
        - clusterrolebindings
    
    
        - clusterroles
    
    
        - cronjobs
    
    
        - daemonsets
    
    
        - deployments
    
    
        - events
    
    
        - flowschemas
    
    
        - horizontalpodautoscalers
    
    
        - ingressclasses
    
    
        - ingresses
    
    
        - jobs
    
    
        - localsubjectaccessreviews
    
    
        - mutatingwebhookconfigurations
    
    
        - namespaces
    
    
        - networkpolicies
    
    
        - nodes
    
    
        - persistentvolumes
    
    
        - poddisruptionbudgets
    
    
        - pods
    
    
        - podsecuritypolicies
    
    
        - podtemplates
    
    
        - replicasets
    
    
        - rolebindings
    
    
        - roles
    
    
        - selfsubjectaccessreviews
    
    
        - selfsubjectrulesreviews
    
    
        - serviceaccounts
    
    
        - services
    
    
        - statefulsets
    
    
        - subjectaccessreviews
    
    
        - tokenreviews
    
    
        - validatingwebhookconfigurations
    
    
        - volumeattachments
    
    
        verbs:
    
    
        - get
     
   
        - list
    

Use kubectl to apply this role:

    =kubectl apply -f <your-expel-role-manifest>  

Next, bind this role to a Workbench user with a ClusterRoleBinding. Create a ClusterRoleBinding manifest:

    
      kind: ClusterRoleBinding
    
    
      apiVersion: rbac.authorization.k8s.io/v1
    
    
      metadata:
    
    
        name: expel-reader-clusterrolebinding
    
    
      subjects:
    
    
      - kind: User
    
    
        name: expel-user
    
    
      roleRef:
    
    
        kind: ClusterRole
    
    
        name: expel-reader-clusterrole
    
    
        apiGroup: rbac.authorization.k8s.io
    

Use kubectl to apply this role:

kubectl apply -f <your-expel-role-binding-manifest>

Finally, map the Workbench IAM role to the in-cluster user “expel-user”. This requires editing the “aws-auth” configMap with EKSctl.

 
      eksctl create iamidentitymapping \
    
        --cluster <your-cluster-name> \
        --region <your-region> \
        --arn <your-expel-role-arn> \
    
          --username expel-user
    

You can confirm the mapping is created by running:

eksctl get iamidentitymapping --cluster <your-cluster-name> --region <your-region>

Step 4: Connecting to Workbench

The steps below outline how to finish connecting to Workbench.

Put these IP addresses in your allow list