Health checks serve as the first line of defense for ensuring data ingestion into Workbench. In most cases, you are responsible for resolving a failed health check that results in an unhealthy device, but there are a few exceptions.

Quick Links

• About Health Checks
  • Types of Health Checks
  • Device Health Monitoring (DHM) Philosophy
  • Health Check Failures
  • Recommended Preventive Device Health Tasks
• View Device Health
• Fix an Unhealthy Device
• Reference
  • Health Statuses You Must Resolve
  • Health Statuses Expel Must Resolve
  • AWS Errors

About Health Checks

Types of Health Checks

There are three types of health checks:

1. API health checks apply to devices with “direct” integrations to Workbench via an API connection.
   • These checks focus on ensuring there is a viable connection between your integration(s) and Workbench so that events can be ingested.
2. Via SIEM health checks apply to devices that are connected to Workbench via a SIEM (because a direct API connection is not available), and there are two checks available.
   • The configuration check looks for the correct credentials and query substitution fields to verify the query syntax.
   • The parent SIEM check looks at the health of the parent SIEM device and updates the status of any child devices accordingly.
3. Device Health Monitoring (DHM) health checks are done by Workbench to identify possible ingestion anomalies.
   • DHM leverages statistical forecasting.
   • See Device Health Monitoring (DHM) Philosophy to learn more about how this works.

Note

Health checks are not available for collector devices, and this will be reflected in the status shown for those devices.

Device Health Monitoring (DHM) Philosophy

When we are monitoring the health of your device, we are not just looking for gaps in connectivity (these are your API and via SIEM health checks) but also evaluating the overall pattern of data ingestion behavior. Our DHM health checks apply time-series forecasting models (Prophet and ARIMA) to determine what “healthy” or “normal” looks like for each device.

When a possible ingestion anomaly is identified, we triage the anomaly and work to determine if the model produced a false positive or a true positive. We will reach out if we identify a true positive. 

Health Check Failures

If any health check fails, the status of the device (and any via SIEM child devices) immediately changes to unhealthy. You will also automatically be alerted to a change in device status via your default Workbench notifications (to a channel like Slack, Teams, Webhooks, etc.) if you have set up the integration for your organization. Email notifications can be enabled as well.

Because device health is part of your default Workbench notifications, it is not necessary to manually check Workbench for a device health status change after you have enabled organization notifications. You will receive an alert from Ruxie that looks like this:

Note

For via SIEM devices, the child device will become unhealthy if the parent device becomes unhealthy (and it will return to a healthy status only when the parent device is healthy). For example, if a Zscaler (via SIEM) device is connected to Workbench through a Splunk device, and the Splunk device becomes unhealthy, then the Zscaler (via SIEM) device will also become unhealthy.

Recommended Preventive Device Health Tasks

While Workbench can detect when data ingestion stops, it cannot determine the root cause. To prevent unnecessary interruptions to data ingestion and the resulting unhealthy device status, we recommend you:

• Monitor for planned maintenance or configuration changes that might affect data ingestion
• Inform Expel of any scheduled downtime

View Device Health

There are multiple ways to view the health of your device. The Security Devices page (Organization Settings > Security Devices) is one of them, but you can also use the Alert Analysis Dashboard. 

For a high-level snapshot of a device's connectivity status, use the Status column to check on the connection. If a problem exists, you will see a message here along with a timestamp for when the device became unhealthy.

• Most devices should indicate a Healthy API connection soon after the new device configuration has been saved. If this does not happen for a new device, your first step should be to check your device configuration for errors.
• See the Reference section for a full list of unhealthy statuses. 

Note

You may occasionally see a status of "No alerts mode" which indicates that polling has been intentionally disabled or is not supported, and we are not ingesting data at this time. Contact support if you need help with this status or if it appears unexpectedly on a device.

Fix an Unhealthy Device

If you see an unhealthy message in the status column or receive a notification about an unhealthy device from Ruxie, you will usually need to resolve the issue yourself (there are a few exceptions). For via SIEM devices, you may need to fix the parent device in order to restore the child device to a healthy status.

To fix a newly added device that shows as unhealthy after onboarding:

• Check your device configuration for errors.
• If you are unable to find the issue within your configuration, or if your device becomes healthy but does not begin polling within 15 minutes and receiving data within 30 minutes, contact support for help.

To fix an existing device that was previously healthy, do one of the following:

• Follow the instructions given to you by Ruxie, or
• Go into the device details (see View Security Device Details for more information about this screen and how to access it) and look for a matching "unhealthy" banner, then select the small arrow to display the steps to fix. For via SIEM devices, you may need to also check the parent device.

Note

See the Reference for a full list of health statuses, error messages, and the general actions required to resolve each one. 

Reference

Health Statuses You Must Resolve

The following chart shows some specific definitions for unhealthy statuses that must be resolved by you, and the required action(s) to take.

Health Statuses Expel Must Resolve

The following statuses must be resolved by Expel.

AWS Errors

AWS users may also see the following errors as a result of health checks, and these errors must be resolved by you. The error itself will show in the Device Details within "Steps to Fix."