The Expel Amazon Elastic Kubernetes Service (EKS) consumes audit logs from the AWS platform through Kinesis. This visibility allows Workbench to identify activity of interest in EKS, investigate, and notify organizations if action is recommended.
Complete the steps below for each EKS cluster in each AWS account to be monitored by Workbench.
To complete this procedure you need:
-
Administrative access to Amazon Web Services (AWS).
-
User privileges to create a Kinesis stream, CloudWatch log subscription, IAM role, and IAM policy.
Enable control plane logging for each EKS cluster
At a minimum, Workbench requires logging of the audit events for each cluster. This provides visibility into activity affecting resources in the cluster (the “who, what, and when” we need to detect and respond to).
Update logging configuration with the AWS CLI like this:
aws eks update-cluster-config \ --region <your-cluster-region> \ --name <your-cluster-name> \ --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'
For more information on configuring control plane logging, see the Amazon reference guide.
Send EKS logs to a Kinesis stream
After control plane logging is enabled, EKS begins sending logs to CloudWatch. Next, the logs must be routed from CloudWatch to a Kinesis data stream so that Workbench can consume the logs in real time. Complete the steps below to route EKS logs to a Kinesis data stream.
Step 1: Create a new Kinesis data stream
Create a stream with the AWS CLI like this:
aws kinesis create-stream \--region <your-cluster-region> \
--stream-name <your-stream-name> \
--stream-mode-details '{"StreamMode": "ON_DEMAND"}'
We recommend creating an “ON_DEMAND”
stream to allow the stream to adjust capacity based on demand.
Step 2: Create an IAM role for CloudWatch log delivery
An IAM role is required to allow CloudWatch to deliver logs to your Kinesis stream.
To start, create a trust policy document that allows the CloudWatch service to assume the role:
{"Statement": {
"Effect": "Allow",
"Principal": { "Service": "logs. <your-region> .amazonaws.com" },
"Action": "sts:AssumeRole",
"Condition": {
"StringLike": { "aws:SourceArn": "arn:aws:logs: <your-region> : <your-account-id> :*" }
}
}
}
Then, create the IAM role supplying this trust document:
aws iam create-role \
--role-name <your-cloudwatch-role-name>
--assume-role-policy-document file:// <your-trust-document-file-path>
Next, create an IAM policy document that grants this role permissions to put records into your Kinesis stream:
{
"Statement": [
{
"Effect": "Allow",
"Action": "kinesis:PutRecord",
"Resource": " <your-kinesis-arn> "
}
]
}
Finally, apply this IAM policy to the CloudWatch role:
aws iam put-role-policy \
--role-name <your-cloudwatch-role-name> \ --policy-name “<your-policy-name>” \--policy-document file://<your-policy-document-path>
Step 3: Route logs from CloudWatch to Kinesis stream
Logs can be routed to Kinesis with a CloudWatch log group subscription filter. Create this filter with the AWS CLI like this:
aws logs put-subscription-filter \
--region <your-cluster-region> \
–log-group-name <your-cluster-log-group-name> \
–filter-name “AllEKSLogs” \
–filter-pattern “”
–destination-arn <your-kinesis-stream-arn>
--role-arn <your-cloudwatch-role>
Create IAM Role for Workbench
To authenticate to your AWS account and retrieve logs, Workbench requires an IAM role. If you already have an Workbench IAM role for another Workbench integration (like CloudWatch or AWS GuardDuty), this existing role can be reused and you can go to: Grant Expel IAM Role required permissions.
To start, create a trust policy document that allows Workbench to assume the role:
{
"Statement":
[
{
"Effect": "Allow",
"Principal":
{
"AWS": "arn:aws:iam::012205512454:user/ExpelCloudService"
},
"Action": "sts:AssumeRole",
"Condition":
{
"StringEquals":
{
"sts:ExternalId": " <your-customer-guid> "
}
}
}
]
}
Then, create the IAM role supplying this trust document:
aws iam create-role \
--role-name <your-expel-role-name>
--assume-role-policy-document file:// <your-trust-document-file-path>
Grant Workbench IAM Role required permissions
Workbench requires IAM permissions to retrieve logs from Kinesis, as well as investigate activity of interest affecting your EKS clusters.
First, create an IAM policy document granting Workbench required permissions for Kinesis and EKS:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:DescribeLimits",
"kinesis:DescribeStream",
"kinesis:DescribeStreamSummary",
"kinesis:GetRecords",
"kinesis:GetShardIterator",
"kinesis:ListShards"
],
"Resource": " <your kinesis stream arn> "
},
{
"Effect": "Allow",
"Action": [
"eks:AccessKubernetesApi",
"eks:DescribeCluster",
"eks:DescribeNodegroup",
"eks:ListClusters",
"eks:ListNodegroups",
"eks:ListUpdates",
"sts:GetCallerIdentity",
"ec2:DescribeRegions",
"autoscaling:DescribeAutoScalingGroups"
],
"Resource": "*"
}
]
}
Then, apply this policy to the Expel IAM role:
aws iam put-role-policy \
--role-name <your-expel-role-name> \
--policy-name “<your-policy-name>” \
--policy-document file://<your-policy-document-path>
Grant Workbench read-only cluster permissions
Expel suggests providing limited read-only access to your EKS clusters to enable deep investigation of interesting activity. These permissions can be fine-tuned based on your needs. Workbench can deliver service without this access, but it significantly limits our ability to thoroughly investigate activity in your clusters.
First, create an Expel ClusterRole manifest to grant limited, read-only permissions:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: expel-reader-clusterrole
rules:
- apiGroups:
- ""
- admissionregistration.k8s.io
- apps
- networking.k8s.io
- rbac.authorization.k8s.io
resources:
- apiservices
- clusterrolebindings
- clusterroles
- cronjobs
- daemonsets
- deployments
- events
- flowschemas
- horizontalpodautoscalers
- ingressclasses
- ingresses
- jobs
- localsubjectaccessreviews
- mutatingwebhookconfigurations
- namespaces
- networkpolicies
- nodes
- persistentvolumes
- poddisruptionbudgets
- pods
- podsecuritypolicies
- podtemplates
- replicasets
- rolebindings
- roles
- selfsubjectaccessreviews
- selfsubjectrulesreviews
- serviceaccounts
- services
- statefulsets
- subjectaccessreviews
- tokenreviews
- validatingwebhookconfigurations
- volumeattachments
verbs:
- get
- list
Use kubectl to apply this role:
=kubectl apply -f <your-expel-role-manifest>
Next, bind this role to a Workbench user with a ClusterRoleBinding. Create a ClusterRoleBinding manifest:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: expel-reader-clusterrolebinding
subjects:
- kind: User
name: expel-user
roleRef:
kind: ClusterRole
name: expel-reader-clusterrole
apiGroup: rbac.authorization.k8s.io
Use kubectl to apply this role:
kubectl apply -f <your-expel-role-binding-manifest>
Finally, map the Workbench IAM role to the in-cluster user “expel-user”. This requires editing the “aws-auth” configMap with EKSctl.
eksctl create iamidentitymapping \
--cluster <your-cluster-name> \
--region <your-region> \
--arn <your-expel-role-arn> \
--username expel-user
You can confirm the mapping is created by running:
eksctl get iamidentitymapping --cluster <your-cluster-name> --region <your-region>
Step 4: Connecting to Workbench
The steps below outline how to finish connecting to Workbench.
-
Log in to Workbench at https://workbench.expel.io.
-
Navigate to Settings > Security devices.
-
Click +Add security device.
-
Select Amazon Elastic Kubernetes Service (EKS).
-
Name the device, provide a description, and fill in the Role ARN, Role Session Name, Region, External ID, and Kinesis Stream Name.
Comments
0 comments
Article is closed for comments.