The Expel Google Kubernetes Engine (GKE) integration consumes audit logs from the Google Cloud Platform (GCP) through a log sink and pub/sub, which allows Workbench to identify activities of interest in GKE, investigate issues, and notify organizations if they need to take action or remediate.
In this article
Resources and permissions
Make sure you have the permissions required to create the following resources:
|
Resource |
Required permissions |
Purpose |
|---|---|---|
|
Log sink |
|
MDR |
|
Workbench GCP Service Account |
|
MDR |
|
Google pub/sub |
|
MDR |
|
Cloud infra custom role |
|
Inventory/metrics |
Configuring MDR
When configuring MDR, use the following checklist to track your progress:
Enable control plane logging for GKE clusters
By default, GKE clusters are configured to send Kubernetes audit events to Cloud Logging. If the default state of clusters isn't changed, no additional configuration is needed.
Otherwise, at a minimum, Workbench requires logging of the Kubernetes control plane audit events for each cluster. Tracking activities that affect resources in the cluster provides necessary information for Expel: the “who”, “what”, and “when” that we need to detect, and take action.
Enable Cloud Resource Manager API for monitored projects
For Expel to discover GKE clusters in the environment, the Cloud Resource Manager API must be enabled in monitored projects.
To enable Cloud Resource Manager API, do the following:
-
In the Google Cloud console, navigate to APIs & Services library
-
From the library, select Cloud Resources Manager, and enable this API for all monitored projects.
Enable Kubernetes Engine API for monitored projects
To enable the Kubernetes Engine API, do the following:
- In the Google Cloud console, navigate to APIs & Services library.
- From the library, select Kubernetes Engine API.
- Enable this API for all monitored projects.
Enable data access auditing
Google Kubernetes Engine (GKE) separates the Kubernetes control plane logs into the following log types in Cloud Logging:
-
Admin Activity logs
Capture write operations, like GET, UPDATE, or DELETE.
-
Optional Data Access logs
Capture read operations, like GET or LIST.
For more information, see the GKE documentation.
Note
Admin Activity logging is enabled by default. To increase logging visibility and include read operations, you may need to enable Data Access logging.
Note
You can enable auditing at a project, folder, or organization level.
To enable Data Access logging, do the following:
-
Navigate to the IAM > Audit Logs page, and search for Kubernetes Engine API.
-
Select the following three boxes as in the figure below:
-
Admin Read
-
Data Read
-
Data Write
Note
Enabling additional logging can increase costs for Cloud Logging. For more information on how this is priced, see the Google Cloud documentation. -
Send GKE logs to a pub/sub topic
GKE begins routing Kubernetes logs to Cloud Logging automatically. Next, we create a pub/sub queue and a log sink to route logs to Workbench.
Step 1: Create a new pub/sub queue
Note
Note the topic and subscription paths for later use.
Make sure the topic path has the following structure:
projects/<project id>/topics/expel-k8s-integration-topic
Workbench collects GKE logs through a pub/sub subscription. You can create a subscription in the web console UI or in the command line interface (CLI) with gcloud.
Option 1: Use the CLI
To create a pub/sub subscription, do the following:
export SUBSCRIPTION_ID=<your subscription id> export TOPIC_ID=<your topic id - save this for later> gcloud pubsub topics create $TOPIC_ID gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic=$TOPIC_ID
Option 2: Use the web console UI
To create a pub/sub subscription, do the following:
-
Navigate to Pub/Sub > Topics.
-
Create a new topic with a default subscription, as in the following figure.
We don't recommend enabling retention duration because of additional cost. If it's needed, you can configure a customer-managed encryption key.
Step 2: Create a log sink to route logs to pub/sub
Steps for adding the log sink are different for the GCP Organization level and for the GCP Project level. Depending on your environmental needs, you have the following options:
Option 1: Create an Organization-level log sink
Create a log sink to route logs to the pub/sub topic. To create a sink at the Organization level, you need to use the gcloud command line utility.
Note
The Organization level is differentiated from the Project level by adding the --include-children and --organization=[org-id] parameters to the create sink command.
To create an Organization-level log sink, do the following:
-
Log in to GCP:
$ gcloud auth login -
Note the GCP org ID from the output of
gcloud organizations list:$ gcloud organizations list DISPLAY_NAME ID DIRECTORY_CUSTOMER_ID <your org> <your org id> <customer id>
-
Create the sink:
export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink> export ORG_ID=<your org id> export TOPIC_ID=expel-k8s-integration-topic gcloud logging sinks create ${EXPEL_LOG_SINK_NAME} \ pubsub.googleapis.com/projects/${EXPEL_LOG_SINK_PROJECT_ID}/topics/${TOPIC_ID} \ --project=${EXPEL_LOG_SINK_PROJECT_ID} \ --include-children --organization=${ORG_ID} \ --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\" -proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\" -proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\" -proto_payload.method_name=\"io.k8s.autoscaling\""$ gcloud logging sinks create [name of log sink] pubsub.googleapis.com/projects/[project-id]/topics/[topic-id] --include-children --organization=[org-id] --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster)\n-proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\"\n-proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\"\n-proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\"\n-proto_payload.method_name=\"io.k8s.autoscaling\""
-
Confirm the sink is successfully created:
$ export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink> $ export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink $ gcloud logging sinks list --project $EXPEL_LOG_SINK_PROJECT_ID | grep $EXPEL_LOG_SINK_NAME NAME DESTINATION FILTER [...] expel-k8s-log-sink pubsub.googleapis.com/projects/<redacted>/topics/expel-k8s-integration-topic (resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name="io.k8s.core.v1.nodes.proxy.get" -proto_payload.method_name="io.k8s.coordination.v1.leases.update" -proto_payload.method_name="io.k8s.core.v1.limitranges.update" -proto_payload.method_name="io.k8s.autoscaling" [...]
Option 2: Create a Project-level log sink
Create a log sink to route logs to the pub/sub topic.
You can use the command line interface or the web console UI:
Option 1: Use the CLI
To create a sink at the Project level, in the gcloud command line utility, do the following:
-
Log in to GCP:
$ gcloud auth login -
Note the GCP org ID from the output of
gcloud organizations list:$ gcloud organizations list DISPLAY_NAME ID DIRECTORY_CUSTOMER_ID <your org> <your org id> <customer id>
-
Create the sink:
export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink> export ORG_ID=<your org id> export TOPIC_ID=expel-k8s-integration-topic gcloud logging sinks create ${EXPEL_LOG_SINK_NAME} \ pubsub.googleapis.com/projects/${EXPEL_LOG_SINK_PROJECT_ID}/topics/${TOPIC_ID} \ --project=${EXPEL_LOG_SINK_PROJECT_ID} \ --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\" -proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\" -proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\" -proto_payload.method_name=\"io.k8s.autoscaling\""
-
Confirm the sink is successfully created:
$ export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink> $ export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink $ gcloud logging sinks list --project $EXPEL_LOG_SINK_PROJECT_ID | grep $EXPEL_LOG_SINK_NAME NAME DESTINATION FILTER [...] expel-k8s-log-sink pubsub.googleapis.com/projects/<redacted>/topics/expel-k8s-integration-topic (resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name="io.k8s.core.v1.nodes.proxy.get" -proto_payload.method_name="io.k8s.coordination.v1.leases.update" -proto_payload.method_name="io.k8s.core.v1.limitranges.update" -proto_payload.method_name="io.k8s.autoscaling" [...]
Option 2: Use the Web console UI
-
Navigate to Cloud Logging > Logs Router and create a new sink.
-
Complete the fields:
In this field...
put this information...
Name
A name for the sink.
We recommend
expel-k8s-log-sink.Description
A description for the sink.
We recommend
GKE logs for Expel monitoring.Destination
The Cloud pub/sub topic created in Step 1:
projects/${EXPEL_LOG_SINK_PROJECT_ID}/topics/expel-k8s-integration-topicBuild inclusion filter
resource.type=gke_cluster OR resource.type=k8s_cluster
Enable Workbench access
Create IAM role for Workbench Kubernetes access
Create a new custom IAM role to capture the required permissions for Workbench.
You can use the web console UI or the command line interface:
Option 1: Use the CLI
To create a custom IAM role, do the following:
export EXPEL_K8S_RO_ROLE=ExpelK8sReadOnlyRole
export K8S_CLUSTER_PROJECT=<your cluster project>
gcloud iam roles create ${EXPEL_K8S_RO_ROLE} --project=${K8S_CLUSTER_PROJECT} --permissions=compute.instanceGroupManagers.get,container.apiServices.get,container.apiServices.getStatus,container.apiServices.list,container.auditSinks.get,container.auditSinks.list,container.backendConfigs.get,container.backendConfigs.list,container.bindings.get,container.bindings.list,container.certificateSigningRequests.get,container.certificateSigningRequests.getStatus,container.certificateSigningRequests.list,container.clusterRoleBindings.get,container.clusterRoleBindings.list,container.clusterRoles.get,container.clusterRoles.list,container.clusters.get,container.clusters.list,container.componentStatuses.get,container.componentStatuses.list,container.controllerRevisions.get,container.controllerRevisions.list,container.cronJobs.get,container.cronJobs.getStatus,container.cronJobs.list,container.csiDrivers.get,container.csiDrivers.list,container.csiNodeInfos.get,container.csiNodeInfos.list,container.csiNodes.get,container.csiNodes.list,container.customResourceDefinitions.get,container.customResourceDefinitions.getStatus,container.customResourceDefinitions.list,container.daemonSets.get,container.daemonSets.getStatus,container.daemonSets.list,container.deployments.get,container.deployments.getScale,container.deployments.getStatus,container.deployments.list,container.endpointSlices.get,container.endpointSlices.list,container.endpoints.get,container.endpoints.list,container.events.get,container.events.list,container.frontendConfigs.get,container.frontendConfigs.list,container.horizontalPodAutoscalers.get,container.horizontalPodAutoscalers.getStatus,container.horizontalPodAutoscalers.list,container.ingresses.get,container.ingresses.getStatus,container.ingresses.list,container.initializerConfigurations.get,container.initializerConfigurations.list,container.jobs.get,container.jobs.getStatus,container.jobs.list,container.mutatingWebhookConfigurations.get,container.mutatingWebhookConfigurations.list,container.namespaces.get,container.namespaces.getStatus,container.namespaces.list,container.networkPolicies.get,container.networkPolicies.list,container.nodes.get,container.nodes.getStatus,container.nodes.list,container.operations.get,container.operations.list,container.persistentVolumeClaims.get,container.persistentVolumeClaims.getStatus,container.persistentVolumeClaims.list,container.persistentVolumes.get,container.persistentVolumes.getStatus,container.persistentVolumes.list,container.petSets.get,container.petSets.list,container.podDisruptionBudgets.get,container.podDisruptionBudgets.getStatus,container.podDisruptionBudgets.list,container.podPresets.get,container.podPresets.list,container.podSecurityPolicies.get,container.podSecurityPolicies.list,container.podTemplates.get,container.podTemplates.list,container.pods.get,container.pods.getStatus,container.pods.list,container.priorityClasses.get,container.priorityClasses.list,container.replicaSets.get,container.replicaSets.getScale,container.replicaSets.getStatus,container.replicaSets.list,container.replicationControllers.get,container.replicationControllers.getScale,container.replicationControllers.getStatus,container.replicationControllers.list,container.resourceQuotas.get,container.resourceQuotas.getStatus,container.resourceQuotas.list,container.roleBindings.get,container.roleBindings.list,container.roles.get,container.roles.list,container.runtimeClasses.get,container.runtimeClasses.list,container.scheduledJobs.get,container.scheduledJobs.list,container.serviceAccounts.get,container.serviceAccounts.list,container.services.get,container.services.getStatus,container.services.list,container.statefulSets.get,container.statefulSets.getScale,container.statefulSets.getStatus,container.statefulSets.list,container.storageClasses.get,container.storageClasses.list,container.storageStates.get,container.storageStates.getStatus,container.storageStates.list,container.storageVersionMigrations.get,container.storageVersionMigrations.getStatus,container.storageVersionMigrations.list,container.thirdPartyObjects.get,container.thirdPartyObjects.list,container.thirdPartyResources.get,container.thirdPartyResources.list,container.tokenReviews.create,container.updateInfos.get,container.updateInfos.list,container.validatingWebhookConfigurations.get,container.validatingWebhookConfigurations.list,container.volumeAttachments.get,container.volumeAttachments.getStatus,container.volumeAttachments.list,container.volumeSnapshotClasses.get,container.volumeSnapshotClasses.list,container.volumeSnapshotContents.get,container.volumeSnapshotContents.getStatus,container.volumeSnapshotContents.list,container.volumeSnapshots.get,container.volumeSnapshots.list
Note
After you run the command, CLI generates a warning about some of these permissions being in testing. You can safely ignore this warning at this time. The following is an example of the warning:
Note: permissions [container.bindings.get, container.bindings.list, container.initializerConfigurations.get, container.initializerConfigurations.list, container.petSets.get, container.petSets.list, container.podPresets.get, container.podPresets.list, container.scheduledJobs.get, container.scheduledJobs.list, container.thirdPartyResources.get, container.thirdPartyResources.list] are in 'TESTING' stage, which means the functionality is not mature and they can go away in the future. This can break your workflows, so do not use them in production systems!
Option 2: Use the Web console UI
To create a custom IAM role, do the following:
-
Navigate to IAM > Roles, and create a new role.
-
Click the ADD PERMISSIONS button, and add the following permissions to this role:
Create a GCP service account for Workbench
Create a service account for Workbench to collect GKE logs and monitor GKE clusters.
To create a service account, do the following:
-
Navigate to IAM > Service Accounts, and create a new service account.
-
Under Service account details, complete the following fields with your information:
-
Service account name: for example,
expel-integration-gke-account. -
Service account ID: for example,
expel-integration-gke-account. -
Service account description: for example,
Expel GKE Integration Service Account.
-
-
Click Create and Continue.
-
Add the following roles to the service account:
-
The Browser role.
This allows Workbench to discover GKE clusters in the environment.
-
The Pub/Sub Subscriber role.
Note
We recommend adding a condition to limit the Custom GKE Reader role to the pub/resource you created earlier. -
-
For the last section, leave the blank defaults, and click Done.
Note
To confirm that these roles are assigned correctly to the service account, run the following commands:
$ export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
$ export EXPEL_K8S_SA_NAME=<the service account name>
$ export EXPEL_K8S_SA="${EXPEL_K8S_SA_NAME}@${EXPEL_LOG_SINK_PROJECT_ID}.iam.gserviceaccount.com"
$ gcloud projects get-iam-policy ${EXPEL_LOG_SINK_PROJECT_ID} \
--flatten="bindings[].members" \
--format='table(bindings.role)' \
--filter="bindings.members:${EXPEL_K8S_SA}"
ROLE
projects/expel-engineering-lab/roles/ExpelIntegrationKubernetesReader
roles/browser
roles/pubsub.subscriber
Onboard to Workbench
The Expel Workbench connection requires the following:
-
The service account key for the service account you created.
The key is a JSON blob starting with
"type": "service-account". -
The full subscription name for the topic you created.
For example,
projects/<project id>/subscriptions/<subcription id>. -
The 12-digit numeric organization (or project) ID for the project for monitoring.
Download the GCP service account key
Workbench requires a service account key to authenticate to your GCP environment.
To download the GCP service account key:
-
Navigate to IAM > Service Accounts.
-
Locate the service account you created earlier for Workbench.
-
From the Actions menu for that account, select Manage Keys.
-
Create a new key.
-
Download the key in the default JSON format.
This key is used in Workbench to onboard the GKE integration.
Configure Expel Workbench
To configure Expel Workbench, do the following:
-
At https://workbench.expel.io, log in to Workbench.
-
Navigate to Settings > Security devices.
-
Click + Add security device.
-
In the search field, type
GKE. -
Select Google Kubernetes Engine.
-
Type the following information:
-
Name
-
Location
-
Service account JSON
This is the GCP service account key.
-
Subscription name
This is the full subscription name for the topic you created.
Example:
projects/<project id>/subscriptions/<subcription id> -
Organization ID
-
-
Click Save.
If all steps are successful, the device shows the Healthy API connection status.