Identity Classification is a type of Investigative Action that leverages a machine learning model to help quickly assess incoming identity alerts. It allows our analysts to more efficiently perform initial triage so that they can spend more of their time on the investigation and response.
This technology will only run when there is a qualifying identity alert with high quality log data (more on this below). It will show up in Workbench as an Investigative Action that is titled “Identity Classification” and has an "Expel AI" label.
Quick Links
- What Qualifies
- Model Training
- Assessment Criteria
- Alert Classifications
- Quality Control
- View Your Auto-Closed Identity Alerts
What Qualifies
To qualify as an identity alert, the event must be associated with a SaaS application that contains authentication behaviors, events, and user activity within the cloud. This is defined as having log data from one of the following platforms:
- Office365
- Okta
- Duo
- G Suite
Model Training
The model is trained on a full year of historical cloud identity alerts, learning from past analyst decisions about which alerts were benign or malicious. By studying these outcomes, the model identifies the patterns and features that most strongly predict whether an identity alert is safe.
Assessment Criteria
The model relies on two main types of features to assess alerts.
Prevalence Features
These features measure how common or rare certain activities, locations, or entities are for a user or within an environment. High prevalence (i.e., repeated, consistent activity) typically suggests normal or expected behavior, while low prevalence (i.e., rare or first-time activity) may indicate something unusual or risky.
Examples:
- How often a user logs in from a particular IP address, region, or country
- How frequently a VPN or hosting provider is used
Behavioral Features
These features capture user behaviors or variations in system activity. They help the model distinguish between routine activity and actions that might be suspicious.
Examples:
- The volume and type(s) of actions performed (such as multiple account changes or file operations)
- Recent account or MFA registrations
- The presence of known suspicious behaviors (like use of a new device or user agent)
Alert Classifications
The model assigns one of the following classifications to indicate how strongly an identity alert leans benign or malicious.
| Benign | The model detected clear signals of safe behavior, with 97% or higher confidence. |
| Likely Benign | The model leans safe, but there is a small chance of risk. |
| Inconclusive | The model is uncertain and cannot make a strong call. |
| Suspicious | The model leans toward malicious behavior, but it is not definitive. |
| Malicious | The model detected strong indicators of a threat. |
These classifications are meant to guide the investigation and to show you how confident the model is. For identity alerts classified as benign, Expel AI auto-closes them as benign and removes them from the queue.
Note
Alerts classified as likely benign, inconclusive, suspicious, or malicious are not auto-closed.
Quality Control
Expel uses multiple quality control (QC) processes to ensure the technology does what it is supposed to do.
Randomized Auto-Close Exceptions
A subset of randomly selected identity alerts that are classified as benign will not be auto-closed. These alerts are instead redirected to Expel SOC analysts for triage, so that the human triage decision can be compared to the model's triage decision.
Protective Rules
The model contains protective rules to prevent the auto-close of identity alerts when certain post-processing indicators are evident. These mechanisms act as a safety net to ensure important identity alerts are always triaged by an analyst regardless of their assigned classification.
Arthur AI
We use Arthur AI to watch the model in production. We look at the model's inputs and outputs to identify any drifts in the data, which would indicate potential performance degradation, and to monitor precision. We have set up alerts to notify us of any problems.
View Your Auto-Closed Identity Alerts
To check if an identity alert was auto-closed by Expel AI, find the close comment on the alert and check the following:
- Is the alert closed as benign?
- Was the alert closed by Expel AI?
- Does the close comment reference the Identity Classification Investigative Action?
If the above are true, then the alert was auto-closed by Expel AI. You can view more details by looking at the associated Investigative Action.