The Expel Google Kubernetes Engine (GKE) offering consumes audit logs from the GCP platform through a log sink and pub/sub. This visibility allows Workbench to identify activity of interest in GKE, investigate, and notify organizations if action is recommended.

Kub_Google_1_Overview.png

Onboarding with terraform

Expel maintains a terraform module to help organizations easily onboard GKE with Expel. Contact your engagement manager with any questions about this onboarding method. Otherwise, follow the manual instructions below.

Manual onboarding instructions

Complete the steps below for each GCP project to be monitored by Workbench.

To complete this procedure you need:

  • Administrative access to the GCP environment.

  • User privileges to create a service account, IAM role, log sink, a pub/sub topic, and a pub/sub subscription.

Enable control plane logging for GKE clusters

At a minimum, Workbench requires logging of the Kubernetes control plane audit events for each cluster. This provides visibility into activity affecting resources in the cluster (the “who”, “what”, and “when” we need to detect and respond). By default, GKE clusters are configured to send Kubernetes audit events to Cloud Logging, so no additional configuration is required for clusters.

Enable data access auditing for GKE

Google Kubernetes Engine (GKE) separates the Kubernetes control plane logs into 2 log types in Cloud Logging: Admin Activity and Data Access logs. In general, write operations (like get, update, or delete) are captured in the Admin Activity log while read operations (like get or list) are captured in the optional Data Access logs (see the documentation for more specifics).

Admin Activity logging is enabled by default, but it can be necessary to enable Data Access logging to increase logging visibility to include read operations. To do so, go to the IAM > Audit Logs page and search for Kubernetes Engine API.

Select all 3 boxes as below:

Kub_Google_2_EnableDataAccessAuditing.png

This can be done at a project, folder, or organization level.

Note

Enabling additional logging can increase costs for Cloud Logging. For more information on how this is priced, see the documentation.

Send GKE logs to a pub/sub topic

GKE begins routing Kubernetes logs to Cloud Logging automatically. Next, we create a pub/sub queue and a log sink to route logs to Workbench.

Step 1: Create a new pub/sub queue

Workbench collects GKE logs through a pub/sub subscription. Navigate to Pub/Sub > Topics and create a new topic and default subscription.

Note

We do not recommend enabling retention duration because of additional cost. Optionally, configure a Customer-managed encryption key if needed.

Kub_Google_3_CreateNewPubSub.png

Step 2: Create a log sink to route logs to pub/sub

Next, we need to create a log sink to route logs to the Pub/Sub topic. To create a sink at the organization level we need to use the gcloud command line utility.

  1. Login to GCP.

    $ gcloud auth login

  2. List the organizations and take note of the org ID.

    $ gcloud organizations list

  3. Create the pub/sub org sink. Use this command to create the sink:

    $ gcloud logging sinks create [name of log sink] pubsub.googleapis.com/projects/[project-id]/topics/[topic-id] --include-children --organization=[org-id] --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster)\n-proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\"\n-proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\"\n-proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\"\n-proto_payload.method_name=\"io.k8s.autoscaling\""

    This automatically creates a new service account which must be granted Pub/Sub Publisher permissions on the integration’s topic.

  4. Grant Pub/Sub Publisher role to log sink’s writer identity.

    Obtain the sink's writer identity—an email address—from the new sink. Go to the Log Router page, and select View sink details. The writer identity appears in the Sink details panel.

    From the console, navigate to PubSub > Topics > [topic-id]. On the Permissions tab, grant the service account created above the Publisher role.

    Kub_Google_4_GrantPubSubPublisherRole.png
    Kub_Google_5_GrantPubSubPublisherRoleConfirm.png

If you are onboarding a specific project

Next, we need to create a log sink to route logs to the pub/sub topic. Navigate to Cloud Logging > Logs Router and create a new sink.

Type the following information:

  1. A name for the sink.

  2. A description for the sink.

  3. Select the Cloud Pub/Sub topic we created earlier as the destination.

  4. Type the filter below as an inclusion filter.

Filter

Kub_Google_6_CreateLogSink.png

Create IAM Role for required permissions

Next, we create a new custom IAM role to capture the required permissions for Workbench. Navigate to IAM > Roles and create a new role.

Kub_Google_7_CreateIAMRole.png

Click the ADD PERMISSIONS button and add the below permissions to this role:

  • container.apiServices.get

  • container.apiServices.getStatus

  • container.apiServices.list

  • container.auditSinks.get

  • container.auditSinks.list

  • container.backendConfigs.get

  • container.backendConfigs.list

  • container.bindings.get

  • container.bindings.list

  • container.certificateSigningRequests.get

  • container.certificateSigningRequests.getStatus

  • container.certificateSigningRequests.list

  • container.clusterRoleBindings.get

  • container.clusterRoleBindings.list

  • container.clusterRoles.get

  • container.clusterRoles.list

  • container.clusters.get

  • container.clusters.list

  • container.componentStatuses.get

  • container.componentStatuses.list

  • container.controllerRevisions.get

  • container.controllerRevisions.list

  • container.cronJobs.get

  • container.cronJobs.getStatus

  • container.cronJobs.list

  • container.csiDrivers.get

  • container.csiDrivers.list

  • container.csiNodeInfos.get

  • container.csiNodeInfos.list

  • container.csiNodes.get

  • container.csiNodes.list

  • container.customResourceDefinitions.get

  • container.customResourceDefinitions.getStatus

  • container.customResourceDefinitions.list

  • container.daemonSets.get

  • container.daemonSets.getStatus

  • container.daemonSets.list

  • container.deployments.get

  • container.deployments.getScale

  • container.deployments.getStatus

  • container.deployments.list

  • container.endpointSlices.get

  • container.endpointSlices.list

  • container.endpoints.get

  • container.endpoints.list

  • container.events.get

  • container.events.list

  • container.frontendConfigs.get

  • container.frontendConfigs.list

  • container.horizontalPodAutoscalers.get

  • container.horizontalPodAutoscalers.getStatus

  • container.horizontalPodAutoscalers.list

  • container.ingresses.get

  • container.ingresses.getStatus

  • container.ingresses.list

  • container.initializerConfigurations.get

  • container.initializerConfigurations.list

  • container.jobs.get

  • container.jobs.getStatus

  • container.jobs.list

  • container.mutatingWebhookConfigurations.get

  • container.mutatingWebhookConfigurations.list

  • container.namespaces.get

  • container.namespaces.getStatus

  • container.namespaces.list

  • container.networkPolicies.get

  • container.networkPolicies.list

  • container.nodes.get

  • container.nodes.getStatus

  • container.nodes.list

  • container.operations.get

  • container.operations.list

  • container.persistentVolumeClaims.get

  • container.persistentVolumeClaims.getStatus

  • container.persistentVolumeClaims.list

  • container.persistentVolumes.get

  • container.persistentVolumes.getStatus

  • container.persistentVolumes.list

  • container.petSets.get

  • container.petSets.list

  • container.podDisruptionBudgets.get

  • container.podDisruptionBudgets.getStatus

  • container.podDisruptionBudgets.list

  • container.podPresets.get

  • container.podPresets.list

  • container.podSecurityPolicies.get

  • container.podSecurityPolicies.list

  • container.podTemplates.get

  • container.podTemplates.list

  • container.pods.get

  • container.pods.getStatus

  • container.pods.list

  • container.priorityClasses.get

  • container.priorityClasses.list

  • container.replicaSets.get

  • container.replicaSets.getScale

  • container.replicaSets.getStatus

  • container.replicaSets.list

  • container.replicationControllers.get

  • container.replicationControllers.getScale

  • container.replicationControllers.getStatus

  • container.replicationControllers.list

  • container.resourceQuotas.get

  • container.resourceQuotas.getStatus

  • container.resourceQuotas.list

  • container.roleBindings.get

  • container.roleBindings.list

  • container.roles.get

  • container.roles.list

  • container.runtimeClasses.get

  • container.runtimeClasses.list

  • container.scheduledJobs.get

  • container.scheduledJobs.list

  • container.serviceAccounts.get

  • container.serviceAccounts.list

  • container.services.get

  • container.services.getStatus

  • container.services.list

  • container.statefulSets.get

  • container.statefulSets.getScale

  • container.statefulSets.getStatus

  • container.statefulSets.list

  • container.storageClasses.get

  • container.storageClasses.list

  • container.storageStates.get

  • container.storageStates.getStatus

  • container.storageStates.list

  • container.storageVersionMigrations.get

  • container.storageVersionMigrations.getStatus

  • container.storageVersionMigrations.list

  • container.thirdPartyObjects.get

  • container.thirdPartyObjects.list

  • container.thirdPartyResources.get

  • container.thirdPartyResources.list

  • container.tokenReviews.create

  • container.updateInfos.get

  • container.updateInfos.list

  • container.validatingWebhookConfigurations.get

  • container.validatingWebhookConfigurations.list

  • container.volumeAttachments.get

  • container.volumeAttachments.getStatus

  • container.volumeAttachments.list

  • container.volumeSnapshotClasses.get

  • container.volumeSnapshotClasses.list

  • container.volumeSnapshotContents.get

  • container.volumeSnapshotContents.getStatus

  • container.volumeSnapshotContents.list

  • container.volumeSnapshots.get

  • container.volumeSnapshots.list

Create GCP service account for Workbench

Next, we create a service account for Workbench to collect GKE logs and monitor GKE clusters. Navigate to IAM > Service Accounts and create a new service account.

Under service account details, type a Name, ID and Description and click Create and Continue.

Kub_Google_8_CreateServiceAcct.png

Next, add 3 roles to the service account:

  1. The “Browser” role. This allows Workbench to discover GKE clusters in the environment.

  2. The custom Workbench role we created earlier.

  3. The “Pub/Sub” subscriber role.

    Kub_Google_9_AddRolesToServiceAcct.png

Note

We recommend adding a condition to limit this role to the pub/resource created earlier.

Finally, for the last section, leave the blank defaults and click Done.

Kub_Google_10_FinishServiceAcct.png

Download GCP service account key

Workbench requires a service account key to authenticate to your GCP environment. To generate a key, navigate to IAM > Service Accounts. Locate the service account you created earlier for Workbench, and select Manage Keys from the Actions menu for that account.

Kub_Google_11_CreatePrivateKey.png

Create a new key, and download the key in the default JSON format. We use this key later in Workbench to onboard the GKE integration.

Onboarding to Expel Workbench

The steps below outline how to finish onboarding AKS in Workbench.

  1. Log in to Workbench at https://workbench.expel.io.

  2. Navigate to Settings > Security devices.

  3. Click +Add security device.

  4. Select Google Kubernetes Engine.

    Kub_Google_12_AddSecDev.png
  5. Name the device, provide a description, and fill in the Service Account auth JSON, and the pubsub subscription name.