The Expel Google Kubernetes Engine (GKE) integration consumes audit logs from the Google Cloud Platform (GCP) through a log sink and pub/sub, which allows Workbench to identify activities of interest in GKE, investigate issues, and notify organizations if they need to take action or remediate.

An overview diagram on how Expel integrates with Google Kubernetes Engine

Prerequisites

  1. Ensure you have the permissions to create the necessary resources in this process:

Step 1: Enable Control Plane Logging for GKE Clusters

By default, GKE clusters are configured to send Kubernetes audit events to Cloud Logging. If the default state of clusters isn't changed, no additional configuration is needed.

Otherwise, at a minimum, Workbench requires logging of the Kubernetes control plane audit events for each cluster. Tracking activities that affect resources in the cluster provides necessary information for Expel: the “who”, “what”, and “when” that we need to detect, and take action.

Step 2: Enable Cloud Resource Manager API for Monitored Projects

For Expel to discover GKE clusters in the environment, the Cloud Resource Manager API must be enabled in monitored projects.

To enable Cloud Resource Manager API:

  1. In the Google Cloud console, navigate to APIs & Services library

  2. From the library, select Cloud Resources Manager, and enable this API for all monitored projects.

A screenshot from Cloud Resource Manager API with API Enabled ticked

Step 3: Enable Kubernetes Engine API for Monitored Projects

For Expel to discover GKE clusters and get details of the clusters in the environment, the Kubernetes Engine API must be enabled in monitored projects. This is a must for inventory/metrics and security benchmarking, and a good-to-have for MDR investigations.

To enable the Kubernetes Engine API:

  1. In the Google Cloud console, navigate to APIs & Services library.
  2. From the library, select Kubernetes Engine API.
  3. Enable this API for all monitored projects.

Step 4: Enable Data Access Auditing

Google Kubernetes Engine (GKE) separates the Kubernetes control plane logs into the following log types in Cloud Logging:

  • Admin Activity logs

    Capture write operations, like GET, UPDATE, or DELETE.

  • Optional Data Access logs

    Capture read operations, like GET or LIST.

For more information, see the GKE documentation.

Note
Admin Activity logging is enabled by default. To increase logging visibility and include read operations, you may need to enable Data Access logging.

You can enable auditing at a project, folder, or organization level.

To enable Data Access logging:

  1. Navigate to the IAM > Audit Logs page and search for "Kubernetes Engine API". Select these three boxes:

    • Admin Read

    • Data Read

    • Data Write

A screenshot of Data Access audit logs configuration with ticks under Admin Read, Data Read, and Data Write.

Enabling additional logging can increase costs for Cloud Logging. For more information on how this is priced, see the Google Cloud documentation.

Step 5: Send GKE Logs to a Pub/Sub Topic

GKE begins routing Kubernetes logs to Cloud Logging automatically. In this step, we will create a pub/sub queue and a log sink to route logs to Workbench.

1. Create a New Pub/Sub Queue

Workbench collects GKE logs through a pub/sub subscription. You can create a subscription in the command line interface (CLI) with gcloud or the web console UI.

Be sure to:

  • Note the topic and subscription paths for later use.
  • Make sure the topic path has the following structure:
    projects/<project id>/topics/expel-k8s-integration-topic

Use the CLI

To create a pub/sub subscription using the CLI:

export SUBSCRIPTION_ID=<your subscription id>
export TOPIC_ID=<your topic id - save this for later>
gcloud pubsub topics create $TOPIC_ID
gcloud pubsub subscriptions create $SUBSCRIPTION_ID --topic=$TOPIC_ID


Use the web console UI

To create a pub/sub subscription using the web console UI:

  1. Navigate to Pub/Sub > Topics.

  2. Create a new topic with a default subscription:

    • Enter a name in the Topic ID field.
    • Select the Add a default subscription checkbox. The other checkboxes should be left unchecked.
      Note: We don't recommend enabling retention duration because of additional cost. If it's needed, you can configure a customer-managed encryption key.
    • Under Encryption, select Google-managed encryption key.
    • Select Create.
A Create topic dialog with the following options selected: Add a default subscription, Google-managed encryption key

2. Create a Log Sink to Route Logs to Pub/Sub

Steps for adding the log sink are different for the GCP Organization level and for the GCP Project level. Depending on your environmental needs, you have these options:

Option 1: Create an Organization-level log sink

Create a log sink to route logs to the pub/sub topic. To create a sink at the Organization level, you need to use the gcloud command line utility.

Note
The Organization level is differentiated from the Project level by adding the --include-children and --organization=[org-id] parameters to the create sink command.

To create an Organization-level log sink:

  1. Log in to GCP:

    $ gcloud auth login

  2. Note the GCP org ID from the output of gcloud organizations list:

    $ gcloud organizations list
    DISPLAY_NAME             ID  DIRECTORY_CUSTOMER_ID
    <your org>    <your org id>          <customer id>
    
  3. Create the sink:

    export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink
    export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
    export ORG_ID=<your org id>
    export TOPIC_ID=expel-k8s-integration-topic
    gcloud logging sinks create ${EXPEL_LOG_SINK_NAME} \
        pubsub.googleapis.com/projects/${EXPEL_LOG_SINK_PROJECT_ID}/topics/${TOPIC_ID} \
        --project=${EXPEL_LOG_SINK_PROJECT_ID} \
        --include-children --organization=${ORG_ID} \
        --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\" -proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\" -proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\" -proto_payload.method_name=\"io.k8s.autoscaling\""$ gcloud logging sinks create [name of log sink] pubsub.googleapis.com/projects/[project-id]/topics/[topic-id] --include-children --organization=[org-id] --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster)\n-proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\"\n-proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\"\n-proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\"\n-proto_payload.method_name=\"io.k8s.autoscaling\""
  4. Confirm the sink is successfully created:

    $ export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
    $ export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink
    $ gcloud logging sinks list --project $EXPEL_LOG_SINK_PROJECT_ID | grep $EXPEL_LOG_SINK_NAME
    NAME                                DESTINATION                                                                                     FILTER
    [...]
    expel-k8s-log-sink                  pubsub.googleapis.com/projects/<redacted>/topics/expel-k8s-integration-topic                   (resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name="io.k8s.core.v1.nodes.proxy.get" -proto_payload.method_name="io.k8s.coordination.v1.leases.update" -proto_payload.method_name="io.k8s.core.v1.limitranges.update" -proto_payload.method_name="io.k8s.autoscaling"
    [...]
    

Option 2: Create a Project-level log sink

Create a log sink to route logs to the pub/sub topic. You can use the command line interface (CLI) or the web console UI:

Use the CLI

To create a sink at the Project level, in the gcloud command line utility:

  1. Log in to GCP:

    $ gcloud auth login

  2. Note the GCP org ID from the output of gcloud organizations list:

    $ gcloud organizations list
    DISPLAY_NAME             ID  DIRECTORY_CUSTOMER_ID
    <your org>    <your org id>          <customer id>
  3. Create the sink:

    export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink
    export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
    export ORG_ID=<your org id>
    export TOPIC_ID=expel-k8s-integration-topic
    gcloud logging sinks create ${EXPEL_LOG_SINK_NAME} \
        pubsub.googleapis.com/projects/${EXPEL_LOG_SINK_PROJECT_ID}/topics/${TOPIC_ID} \
        --project=${EXPEL_LOG_SINK_PROJECT_ID} \
        --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\" -proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\" -proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\" -proto_payload.method_name=\"io.k8s.autoscaling\""
  4. Confirm the sink is successfully created:

    $ export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
    $ export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink
    $ gcloud logging sinks list --project $EXPEL_LOG_SINK_PROJECT_ID | grep $EXPEL_LOG_SINK_NAME
    NAME                                DESTINATION                                                                                     FILTER
    [...]
    expel-k8s-log-sink                  pubsub.googleapis.com/projects/<redacted>/topics/expel-k8s-integration-topic                   (resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name="io.k8s.core.v1.nodes.proxy.get" -proto_payload.method_name="io.k8s.coordination.v1.leases.update" -proto_payload.method_name="io.k8s.core.v1.limitranges.update" -proto_payload.method_name="io.k8s.autoscaling"
    [...]

Use the Web console UI

  1. Navigate to Cloud Logging > Logs Router and create a new sink.

  2. Complete the fields:

    In this field...

    put this information...

    Name

    A name for the sink.

    We recommend expel-k8s-log-sink.

    Description

    A description for the sink.

    We recommend GKE logs for Expel monitoring.

    Destination

    The Cloud pub/sub topic created in Step 1:

    projects/${EXPEL_LOG_SINK_PROJECT_ID}/topics/expel-k8s-integration-topic

    Build inclusion filter

    resource.type=gke_cluster OR resource.type=k8s_cluster

A screenshot with sample sink field values. The values are like the ones provided in the table above.

Step 6: Enable Workbench Access

1. Create IAM Role for Workbench Kubernetes Access

Create a new custom IAM role to capture the required permissions for Workbench.

You can use the web console UI or the command line interface:

Use the CLI

To create a custom IAM role:

export EXPEL_K8S_RO_ROLE=ExpelK8sReadOnlyRole
export K8S_CLUSTER_PROJECT=<your cluster project>
gcloud iam roles create ${EXPEL_K8S_RO_ROLE} --project=${K8S_CLUSTER_PROJECT} --permissions=compute.instanceGroupManagers.get,container.apiServices.get,container.apiServices.getStatus,container.apiServices.list,container.auditSinks.get,container.auditSinks.list,container.backendConfigs.get,container.backendConfigs.list,container.bindings.get,container.bindings.list,container.certificateSigningRequests.get,container.certificateSigningRequests.getStatus,container.certificateSigningRequests.list,container.clusterRoleBindings.get,container.clusterRoleBindings.list,container.clusterRoles.get,container.clusterRoles.list,container.clusters.get,container.clusters.list,container.componentStatuses.get,container.componentStatuses.list,container.controllerRevisions.get,container.controllerRevisions.list,container.cronJobs.get,container.cronJobs.getStatus,container.cronJobs.list,container.csiDrivers.get,container.csiDrivers.list,container.csiNodeInfos.get,container.csiNodeInfos.list,container.csiNodes.get,container.csiNodes.list,container.customResourceDefinitions.get,container.customResourceDefinitions.getStatus,container.customResourceDefinitions.list,container.daemonSets.get,container.daemonSets.getStatus,container.daemonSets.list,container.deployments.get,container.deployments.getScale,container.deployments.getStatus,container.deployments.list,container.endpointSlices.get,container.endpointSlices.list,container.endpoints.get,container.endpoints.list,container.events.get,container.events.list,container.frontendConfigs.get,container.frontendConfigs.list,container.horizontalPodAutoscalers.get,container.horizontalPodAutoscalers.getStatus,container.horizontalPodAutoscalers.list,container.ingresses.get,container.ingresses.getStatus,container.ingresses.list,container.initializerConfigurations.get,container.initializerConfigurations.list,container.jobs.get,container.jobs.getStatus,container.jobs.list,container.mutatingWebhookConfigurations.get,container.mutatingWebhookConfigurations.list,container.namespaces.get,container.namespaces.getStatus,container.namespaces.list,container.networkPolicies.get,container.networkPolicies.list,container.nodes.get,container.nodes.getStatus,container.nodes.list,container.operations.get,container.operations.list,container.persistentVolumeClaims.get,container.persistentVolumeClaims.getStatus,container.persistentVolumeClaims.list,container.persistentVolumes.get,container.persistentVolumes.getStatus,container.persistentVolumes.list,container.petSets.get,container.petSets.list,container.podDisruptionBudgets.get,container.podDisruptionBudgets.getStatus,container.podDisruptionBudgets.list,container.podPresets.get,container.podPresets.list,container.podSecurityPolicies.get,container.podSecurityPolicies.list,container.podTemplates.get,container.podTemplates.list,container.pods.get,container.pods.getStatus,container.pods.list,container.priorityClasses.get,container.priorityClasses.list,container.replicaSets.get,container.replicaSets.getScale,container.replicaSets.getStatus,container.replicaSets.list,container.replicationControllers.get,container.replicationControllers.getScale,container.replicationControllers.getStatus,container.replicationControllers.list,container.resourceQuotas.get,container.resourceQuotas.getStatus,container.resourceQuotas.list,container.roleBindings.get,container.roleBindings.list,container.roles.get,container.roles.list,container.runtimeClasses.get,container.runtimeClasses.list,container.scheduledJobs.get,container.scheduledJobs.list,container.serviceAccounts.get,container.serviceAccounts.list,container.services.get,container.services.getStatus,container.services.list,container.statefulSets.get,container.statefulSets.getScale,container.statefulSets.getStatus,container.statefulSets.list,container.storageClasses.get,container.storageClasses.list,container.storageStates.get,container.storageStates.getStatus,container.storageStates.list,container.storageVersionMigrations.get,container.storageVersionMigrations.getStatus,container.storageVersionMigrations.list,container.thirdPartyObjects.get,container.thirdPartyObjects.list,container.thirdPartyResources.get,container.thirdPartyResources.list,container.tokenReviews.create,container.updateInfos.get,container.updateInfos.list,container.validatingWebhookConfigurations.get,container.validatingWebhookConfigurations.list,container.volumeAttachments.get,container.volumeAttachments.getStatus,container.volumeAttachments.list,container.volumeSnapshotClasses.get,container.volumeSnapshotClasses.list,container.volumeSnapshotContents.get,container.volumeSnapshotContents.getStatus,container.volumeSnapshotContents.list,container.volumeSnapshots.get,container.volumeSnapshots.list

Note
After you run the command, CLI generates a warning about some of these permissions being in testing. You can safely ignore this warning at this time. An example:

Note: permissions [container.bindings.get, container.bindings.list, container.initializerConfigurations.get, container.initializerConfigurations.list, container.petSets.get, container.petSets.list, container.podPresets.get, container.podPresets.list, container.scheduledJobs.get, container.scheduledJobs.list, container.thirdPartyResources.get, container.thirdPartyResources.list] are in 'TESTING' stage, which means the functionality is not mature and they can go away in the future. This can break your workflows, so do not use them in production systems!

Use the Web console UI

To create a custom IAM role:

  1. Navigate to IAM > Roles, and create a new role.

  2. Click the ADD PERMISSIONS button, and add these permissions to this role:

2. Create a GCP Service Account for Workbench

Create a service account for Workbench to collect GKE logs and monitor GKE clusters.

To create a service account:

  1. Navigate to IAM > Service Accounts, and create a new service account.

  2. Under Service account details, complete the following fields with your information:

    • Service account name - enter "expel-integration-gke-account".

    • Service account ID - enter "expel-integration-gke-account".

    • Service account description - enter "Expel GKE Integration Service Account".

    The Service account details dialog with the fields filled in like described above.
  3. Click Create and Continue.

  4. Add the following roles to the service account:

    • Browser (this allows Workbench to discover GKE clusters in the environment)

    • Pub/Sub Subscriber

    Screenshot of the Service account details dialog with the roles added, as described above.
    Note
    We recommend adding a condition to limit the Custom GKE Reader role to the pub/resource you created earlier.
  5. For the last section, leave the blank defaults, and click Done.


To confirm that these roles are assigned correctly to the service account, run the following commands:

$ export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
$ export EXPEL_K8S_SA_NAME=<the service account name>
$ export EXPEL_K8S_SA="${EXPEL_K8S_SA_NAME}@${EXPEL_LOG_SINK_PROJECT_ID}.iam.gserviceaccount.com"
$ gcloud projects get-iam-policy ${EXPEL_LOG_SINK_PROJECT_ID} \
  --flatten="bindings[].members" \
  --format='table(bindings.role)' \
  --filter="bindings.members:${EXPEL_K8S_SA}"

ROLE
projects/expel-engineering-lab/roles/ExpelIntegrationKubernetesReader
roles/browser
roles/pubsub.subscriber

Step 7: Add Google Kubernetes Engine as a Security Device in Workbench

The Expel Workbench connection requires:

  • The service account key for the service account you created.

    The key is a JSON blob starting with "type": "service-account".

  • The full subscription name for the topic you created.

    For example, projects/<project id>/subscriptions/<subcription id>.

  • The 12-digit numeric organization (or project) ID for the project for monitoring.

1. Download the GCP Service Account Key

Workbench requires a service account key to authenticate to your GCP environment.

To download the GCP service account key:

  1. Navigate to IAM > Service Accounts.

  2. Locate the service account you created earlier for Workbench.

  3. From the Actions menu for that account, select Manage Keys.

  4. Create a new key.

  5. Download the key in the default JSON format.

    This key is used in Workbench to onboard the GKE integration.

2. Configure Expel Workbench

To configure Expel Workbench:

  1. Log in to Workbench.

  2. Navigate to Settings > Security devices.

  3. Click + Add security device.

  4. In the search field, type GKE.

  5. Select Google Kubernetes Engine.

    Add security device dialog with the connection settings fields filled in.
  6. Enter the following information:

    • Name - enter a name that might help you more easily identify this integration, such as “CompanyName <technology>”; this name will display in Workbench under the Name column, and is a text string that you can filter on.

    • Location - enter the location of your integration, for example “cloud;” this is also a text string that you can filter on, so we recommend being consistent with location naming across your Expel integrations.

    • Service account JSON - enter your GCP service account key.

    • Subscription name - enter the full subscription name for the topic you created.

      Example: projects/<project id>/subscriptions/<subcription id>

    • Organization ID - enter your Google Cloud Organization ID.

  7. Click Save.

Your device should be created successfully within a few seconds. A few reminders:

  • After your connection is healthy, it will take some time for your device to begin polling and receiving data.
  • To check on the status, select the downward arrow for your device in the first column and choose View details. You can then scroll to the Connection section to see if your device is fully connected.
  • Polling will happen first; data will be received after that. You must refresh the page to see updates.
  • If your device does not begin polling within 15 minutes, and does not begin receiving data within 30 minutes, contact our support team for help.