The Expel Google Kubernetes Engine (GKE) offering consumes audit logs from the GCP platform through a log sink and pub/sub. This visibility allows Workbench to identify activity of interest in GKE, investigate, and notify organizations if they need to take action and remediate any issues.

An overview diagram on how Expel integrates with Google Kubernetes Engine

Resources and permissions

Make sure you have the permissions required to create the following resources:

Resource

Required permissions

Purpose

Log sink

roles/logging.configWriter

MDR

Workbench GCP Service Account

roles/resourcemanager.projectIamAdmin

MDR

Google pub/sub

roles/pubsub.admin

MDR

Cloud infra custom role

roles/iam.roleAdmin

Inventory/metrics

Custom Kubernetes role

roles/iam.roleAdmin

Security benchmarking

Configuring MDR

When configuring MDR, use the following checklist to track your progress:

Enable control plane logging for GKE clusters

By default, GKE clusters are configured to send Kubernetes audit events to Cloud Logging. If the default state of clusters isn't changed, no additional configuration is needed.

Otherwise, at a minimum, Workbench requires logging of the Kubernetes control plane audit events for each cluster. Tracking activities that affect resources in the cluster provides necessary information for Expel: the “who”, “what”, and “when” that we need to detect, and take action.

Enable Cloud Resource Manager API for monitored projects

For Expel to discover GKE clusters in the environment, the Cloud Resource Manager API must be enabled in monitored projects.

To enable Cloud Resource Manager API, do the following:

  1. In the Google Cloud console, navigate to APIs & Services library

  2. From the library, select Cloud Resources Manager, and enable this API for all monitored projects.

A screenshot from Cloud Resource Manager API with API Enabled ticked

Enable data access auditing

Google Kubernetes Engine (GKE) separates the Kubernetes control plane logs into the following log types in Cloud Logging:

  • Admin Activity logs

    Capture write operations, like GET, UPDATE, or DELETE.

  • Optional Data Access logs

    Capture read operations, like GET or LIST.

For more information, see the GKE documentation.

Note

Admin Activity logging is enabled by default. To increase logging visibility and include read operations, you may need to enable Data Access logging.

Note

You can enable auditing at a project, folder, or organization level.

To enable Data Access logging, do the following:

  1. Navigate to the IAM > Audit Logs page, and search for Kubernetes Engine API.

  2. Select the following three boxes as in the figure below:

    • Admin Read

    • Data Read

    • Data Write

    A screenshot of Data Access audit logs configuration with ticks under Admin Read, Data Read, and Data Write.

Note

Enabling additional logging can increase costs for Cloud Logging. For more information on how this is priced, see the Google Cloud documentation.

Send GKE logs to a pub/sub topic

GKE begins routing Kubernetes logs to Cloud Logging automatically. Next, we create a pub/sub queue and a log sink to route logs to Workbench.

Step 1: Create a new pub/sub queue

Important

Note the topic and subscription paths for later use.

Make sure the topic path has the following structure:

projects/<project id>/topics/expel-k8s-integration-topic

Workbench collects GKE logs through a pub/sub subscription. You can create a subscription in the web console UI or in the command line interface (CLI) with gcloud.

Note

We don't recommend enabling retention duration because of additional cost. If it's needed, you can configure a customer-managed encryption key.

Step 2: Create a log sink to route logs to pub/sub

Steps for adding the log sink are different for the GCP Organization level and for the GCP Project level. Depending on your environmental needs, you have the following options:

Option 1: Create an Organization-level log sink

Create a log sink to route logs to the pub/sub topic. To create a sink at the Organization level, you need to use the gcloud command line utility.

Note

The Organization level is differentiated from the Project level by adding the --include-children and --organization=[org-id] parameters to the create sink command.

To create an Organization-level log sink, do the following:

  1. Log in to GCP:

    $ gcloud auth login

  2. Note the GCP org ID from the output of gcloud organizations list:

    $ gcloud organizations list
    DISPLAY_NAME             ID  DIRECTORY_CUSTOMER_ID
    <your org>    <your org id>          <customer id>
    
  3. Create the sink:

    export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink
    export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
    export ORG_ID=<your org id>
    export TOPIC_ID=expel-k8s-integration-topic
    gcloud logging sinks create ${EXPEL_LOG_SINK_NAME} \
        pubsub.googleapis.com/projects/${PROJECT_ID}/topics/${TOPIC_ID} \
        --project=${PROJECT_ID} \
        --include-children --organization=${ORG_ID} \
        --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\" -proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\" -proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\" -proto_payload.method_name=\"io.k8s.autoscaling\""$ gcloud logging sinks create [name of log sink] pubsub.googleapis.com/projects/[project-id]/topics/[topic-id] --include-children --organization=[org-id] --log-filter="(resource.type=gke_cluster OR resource.type=k8s_cluster)\n-proto_payload.method_name=\"io.k8s.core.v1.nodes.proxy.get\"\n-proto_payload.method_name=\"io.k8s.coordination.v1.leases.update\"\n-proto_payload.method_name=\"io.k8s.core.v1.limitranges.update\"\n-proto_payload.method_name=\"io.k8s.autoscaling\""      
  4. Confirm the sink is successfully created:

    $ export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
    $ export EXPEL_LOG_SINK_NAME=expel-k8s-log-sink
    $ gcloud logging sinks list --project $EXPEL_LOG_SINK_PROJECT | grep $EXPEL_LOG_SINK_NAME
    NAME                                DESTINATION                                                                                     FILTER
    [...]
    expel-k8s-log-sink                  pubsub.googleapis.com/projects/<redacted>/topics/expel-k8s-integration-topic.                   (resource.type=gke_cluster OR resource.type=k8s_cluster) -proto_payload.method_name="io.k8s.core.v1.nodes.proxy.get" -proto_payload.method_name="io.k8s.coordination.v1.leases.update" -proto_payload.method_name="io.k8s.core.v1.limitranges.update" -proto_payload.method_name="io.k8s.autoscaling"
    [...]
    
Option 2: Create a Project-level log sink

Create a log sink to route logs to the pub/sub topic.

You can use the command line interface or the web console UI:

Enable Workbench access

Create IAM role for Workbench Kubernetes access

Create a new custom IAM role to capture the required permissions for Workbench.

You can use the web console UI or the command line interface:

Option 2: Use the Web console UI

To create a custom IAM role, do the following:

  1. Navigate to IAM > Roles, and create a new role.

  2. Click the ADD PERMISSIONS button, and add the following permissions to this role:

Create a GCP service account for Workbench

Create a service account for Workbench to collect GKE logs and monitor GKE clusters.

To create a service account, do the following:

  1. Navigate to IAM > Service Accounts, and create a new service account.

  2. Under Service account details, complete the following fields with your information:

    • Service account name: for example, expel-integration-gke-account.

    • Service account ID: for example, expel-integration-gke-account.

    • Service account description: for example,Expel GKE Integration Service Account.

    The Service account details dialog with the fields filled in like described above.
  3. Click Create and Continue.

  4. Add the following roles to the service account:

    • The Browser role.

      This allows Workbench to discover GKE clusters in the environment.

    • The Pub/Sub Subscriber role.

    • For the Kubernetes security benchmark report only: the Custom GKE Reader role.

    Screenshot of the Service account details dialog with the roles added, as described above.

    Note

    We recommend adding a condition to limit the Custom GKE Reader role to the pub/resource you created earlier.

  5. For the last section, leave the blank defaults, and click Done.

Tip

To confirm that these roles are assigned correctly to the service account, run the following commands:

$ export EXPEL_LOG_SINK_PROJECT_ID=<your project id for the log sink>
$ export EXPEL_K8S_SA="expel-k8s-sa@${EXPEL_LOG_SINK_PROJECT_ID}.iam.gserviceaccount.com"
$ gcloud projects get-iam-policy ${EXPEL_LOG_SINK_PROJECT_ID} \
  --flatten="bindings[].members" \
  --format='table(bindings.role)' \
  --filter="bindings.members:${EXPEL_K8S_SA}"

ROLE
projects/expel-engineering-lab/roles/ExpelIntegrationKubernetesReader
roles/browser
roles/pubsub.subscriber

Enabling Expel Security Benchmark Report

No additional steps are required in GKE.

Onboard to Workbench

The Expel Workbench connection requires the following:

  • The service account key for the service account you created.

    The key is a JSON blob starting with "type": "service-account".

  • The full subscription name for the topic you created.

    For example, projects/<project id>/subscriptions/<subcription id>.

  • The 12-digit numeric organization (or project) ID for the project for monitoring.

Download the GCP service account key

Workbench requires a service account key to authenticate to your GCP environment.

To download the GCP service account key:

  1. Navigate to IAM > Service Accounts.

  2. Locate the service account you created earlier for Workbench.

  3. From the Actions menu for that account, select Manage Keys.

  4. Create a new key.

  5. Download the key in the default JSON format.

    This key is used in Workbench to onboard the GKE integration.

Configure Expel Workbench

To configure Expel Workbench, do the following:

  1. At https://workbench.expel.io, log in to Workbench.

  2. Navigate to Settings > Security devices.

  3. Click + Add security device.

  4. In the search field, type GKE.

  5. Select Google Kubernetes Engine.

    Add security device dialog with the connection settings fields filled in.
  6. Type the following information:

    • Name

    • Location

    • Service account JSON

      This is the GCP service account key.

    • Subscription name

      This is the full subscription name for the topic you created.

      Example: projects/<project id>/subscriptions/<subcription id>

    • Organization ID

  7. Click Save.

    If all steps are successful, the device shows the Healthy API connection status.

    gke-healthy-api.png

Tip

This page was accurate at the time of writing, but changes happen. If you find the instructions are outdated, let us know via your engagement manager or account representative.