The Expel Google Kubernetes Engine (GKE) offering consumes audit logs from the GCP platform through a log sink and pub/sub. This visibility allows Workbench to identify activity of interest in GKE, investigate, and notify organizations if action is recommended.
Onboarding with terraform
Expel maintains a terraform module to help organizations easily onboard GKE with Expel. Contact your engagement manager with any questions about this onboarding method. Otherwise, follow the manual instructions below.
Manual onboarding instructions
Complete the steps below for each GCP project to be monitored by Workbench.
To complete this procedure you need:
-
Administrative access to the GCP environment.
-
User privileges to create a service account, IAM role, log sink, a pub/sub topic, and a pub/sub subscription.
Enable control plane logging for GKE clusters
At a minimum, Workbench requires logging of the Kubernetes control plane audit events for each cluster. This provides visibility into activity affecting resources in the cluster (the “who”, “what”, and “when” we need to detect and respond). By default, GKE clusters are configured to send Kubernetes audit events to Cloud Logging, so no additional configuration is required for clusters.
Enable Cloud Resource Manager API for monitored projects
For Expel to discover GKE clusters in the environment, the Cloud Resource Manager API needs to be enabled in monitored projects. To enable the Cloud Resource Manager API, navigate to APIs & Services library in the Google Cloud console.
From the library, select Cloud Resources Manager, and enable this API for all monitored projects.
Enable data access auditing for GKE
Google Kubernetes Engine (GKE) separates the Kubernetes control plane logs into 2 log types in Cloud Logging: Admin Activity and Data Access logs. In general, write operations (like get, update, or delete) are captured in the Admin Activity log while read operations (like get or list) are captured in the optional Data Access logs (see the documentation for more specifics).
Admin Activity logging is enabled by default, but it can be necessary to enable Data Access logging to increase logging visibility to include read operations. To do so, go to the IAM > Audit Logs page and search for Kubernetes Engine API.
Select all 3 boxes as below:
This can be done at a project, folder, or organization level.
Note
Enabling additional logging can increase costs for Cloud Logging. For more information on how this is priced, see the documentation.
Send GKE logs to a pub/sub topic
GKE begins routing Kubernetes logs to Cloud Logging automatically. Next, we create a pub/sub queue and a log sink to route logs to Workbench.
Step 1: Create a new pub/sub queue
Workbench collects GKE logs through a pub/sub subscription. Navigate to Pub/Sub > Topics and create a new topic and default subscription.
Note
We don't recommend enabling retention duration because of additional cost. Optionally, configure a Customer-managed encryption key if needed.
Step 2: Create a log sink to route logs to pub/sub
Next, we need to create a log sink to route logs to the Pub/Sub topic. To create a sink at the organization level we need to use the gcloud command line utility.
-
Login to GCP.
$ gcloud auth login -
List the organizations and take note of the org ID.
$ gcloud organizations list -
Create the pub/sub org sink. Use this command to create the sink:
$ gcloud logging sinks create [name of log sink] pubsub.googleapis.com/projects/[project-id]/topics/[topic-id] --include-children --organization=[org-id] --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster)\n-proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\"\n-proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\"\n-proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\"\n-proto_payload.method_name=\"io.k8s.autoscaling\""This automatically creates a new service account which must be granted Pub/Sub Publisher permissions on the integration’s topic.
-
Grant Pub/Sub Publisher role to log sink’s writer identity.
Obtain the sink's writer identity—an email address—from the new sink. Go to the Log Router page, and select View sink details. The writer identity appears in the Sink details panel.
From the console, navigate to PubSub > Topics > [topic-id]. On the Permissions tab, grant the service account created above the Publisher role.
If you are onboarding a specific project
Next, we need to create a log sink to route logs to the pub/sub topic. Navigate to Cloud Logging > Logs Router and create a new sink.
Type the following information:
-
A name for the sink.
-
A description for the sink.
-
Select the Cloud Pub/Sub topic we created earlier as the destination.
-
Type the filter below as an inclusion filter.
Filter
Create IAM Role for required permissions
Next, we create a new custom IAM role to capture the required permissions for Workbench. Navigate to IAM > Roles and create a new role.
Click the ADD PERMISSIONS button and add the below permissions to this role:
-
compute.instanceGroupManagers.get
-
container.apiServices.get
-
container.apiServices.getStatus
-
container.apiServices.list
-
container.auditSinks.get
-
container.auditSinks.list
-
container.backendConfigs.get
-
container.backendConfigs.list
-
container.bindings.get
-
container.bindings.list
-
container.certificateSigningRequests.get
-
container.certificateSigningRequests.getStatus
-
container.certificateSigningRequests.list
-
container.clusterRoleBindings.get
-
container.clusterRoleBindings.list
-
container.clusterRoles.get
-
container.clusterRoles.list
-
container.clusters.get
-
container.clusters.list
-
container.componentStatuses.get
-
container.componentStatuses.list
-
container.controllerRevisions.get
-
container.controllerRevisions.list
-
container.cronJobs.get
-
container.cronJobs.getStatus
-
container.cronJobs.list
-
container.csiDrivers.get
-
container.csiDrivers.list
-
container.csiNodeInfos.get
-
container.csiNodeInfos.list
-
container.csiNodes.get
-
container.csiNodes.list
-
container.customResourceDefinitions.get
-
container.customResourceDefinitions.getStatus
-
container.customResourceDefinitions.list
-
container.daemonSets.get
-
container.daemonSets.getStatus
-
container.daemonSets.list
-
container.deployments.get
-
container.deployments.getScale
-
container.deployments.getStatus
-
container.deployments.list
-
container.endpointSlices.get
-
container.endpointSlices.list
-
container.endpoints.get
-
container.endpoints.list
-
container.events.get
-
container.events.list
-
container.frontendConfigs.get
-
container.frontendConfigs.list
-
container.horizontalPodAutoscalers.get
-
container.horizontalPodAutoscalers.getStatus
-
container.horizontalPodAutoscalers.list
-
container.ingresses.get
-
container.ingresses.getStatus
-
container.ingresses.list
-
container.initializerConfigurations.get
-
container.initializerConfigurations.list
-
container.jobs.get
-
container.jobs.getStatus
-
container.jobs.list
-
container.mutatingWebhookConfigurations.get
-
container.mutatingWebhookConfigurations.list
-
container.namespaces.get
-
container.namespaces.getStatus
-
container.namespaces.list
-
container.networkPolicies.get
-
container.networkPolicies.list
-
container.nodes.get
-
container.nodes.getStatus
-
container.nodes.list
-
container.operations.get
-
container.operations.list
-
container.persistentVolumeClaims.get
-
container.persistentVolumeClaims.getStatus
-
container.persistentVolumeClaims.list
-
container.persistentVolumes.get
-
container.persistentVolumes.getStatus
-
container.persistentVolumes.list
-
container.petSets.get
-
container.petSets.list
-
container.podDisruptionBudgets.get
-
container.podDisruptionBudgets.getStatus
-
container.podDisruptionBudgets.list
-
container.podPresets.get
-
container.podPresets.list
-
container.podSecurityPolicies.get
-
container.podSecurityPolicies.list
-
container.podTemplates.get
-
container.podTemplates.list
-
container.pods.get
-
container.pods.getStatus
-
container.pods.list
-
container.priorityClasses.get
-
container.priorityClasses.list
-
container.replicaSets.get
-
container.replicaSets.getScale
-
container.replicaSets.getStatus
-
container.replicaSets.list
-
container.replicationControllers.get
-
container.replicationControllers.getScale
-
container.replicationControllers.getStatus
-
container.replicationControllers.list
-
container.resourceQuotas.get
-
container.resourceQuotas.getStatus
-
container.resourceQuotas.list
-
container.roleBindings.get
-
container.roleBindings.list
-
container.roles.get
-
container.roles.list
-
container.runtimeClasses.get
-
container.runtimeClasses.list
-
container.scheduledJobs.get
-
container.scheduledJobs.list
-
container.serviceAccounts.get
-
container.serviceAccounts.list
-
container.services.get
-
container.services.getStatus
-
container.services.list
-
container.statefulSets.get
-
container.statefulSets.getScale
-
container.statefulSets.getStatus
-
container.statefulSets.list
-
container.storageClasses.get
-
container.storageClasses.list
-
container.storageStates.get
-
container.storageStates.getStatus
-
container.storageStates.list
-
container.storageVersionMigrations.get
-
container.storageVersionMigrations.getStatus
-
container.storageVersionMigrations.list
-
container.thirdPartyObjects.get
-
container.thirdPartyObjects.list
-
container.thirdPartyResources.get
-
container.thirdPartyResources.list
-
container.tokenReviews.create
-
container.updateInfos.get
-
container.updateInfos.list
-
container.validatingWebhookConfigurations.get
-
container.validatingWebhookConfigurations.list
-
container.volumeAttachments.get
-
container.volumeAttachments.getStatus
-
container.volumeAttachments.list
-
container.volumeSnapshotClasses.get
-
container.volumeSnapshotClasses.list
-
container.volumeSnapshotContents.get
-
container.volumeSnapshotContents.getStatus
-
container.volumeSnapshotContents.list
-
container.volumeSnapshots.get
-
container.volumeSnapshots.list
Create GCP service account for Workbench
Next, we create a service account for Workbench to collect GKE logs and monitor GKE clusters. Navigate to IAM > Service Accounts and create a new service account.
Under service account details, type a Name, ID and Description and click Create and Continue.
Next, add 3 roles to the service account:
-
The “Browser” role. This allows Workbench to discover GKE clusters in the environment.
-
The custom Workbench role we created earlier.
-
The “Pub/Sub” subscriber role.
Note
We recommend adding a condition to limit this role to the pub/resource created earlier.
Finally, for the last section, leave the blank defaults and click Done.
Download GCP service account key
Workbench requires a service account key to authenticate to your GCP environment. To generate a key, navigate to IAM > Service Accounts. Locate the service account you created earlier for Workbench, and select Manage Keys from the Actions menu for that account.
Create a new key, and download the key in the default JSON format. We use this key later in Workbench to onboard the GKE integration.
Onboarding to Expel Workbench
The steps below outline how to finish onboarding in Workbench.
-
Log in to Workbench at https://workbench.expel.io.
-
Navigate to Settings > Security devices.
-
Click + Add security device.
-
Select Google Kubernetes Engine.
-
Name the device, provide a description, and fill in the Service Account auth JSON, and the pubsub subscription name.
Comments
0 comments
Article is closed for comments.