CrowdStrike's AI Triage Research: How Well Can AI Automatically Judge SOC Alerts?
1.
Basic Information Article Title: Teaching AI to Reason Through Detection Triage Publisher: CrowdStrike Publication Date: 2026-08-17 Severity/Importance: High (Practical value as defense research) Original Source: CrowdStrike Related Source: Research Paper (arXiv PDF) Target: Security alerts generated on Windows endpoints Main Models & Methods: Nemotron-3-Nano-30B, Nemotron-3-Super-120B, GEPA, AdaSTaR, LoRA, GRPO
2.
Summary in One Sentence This is research on having AI judge whether Windows endpoint alerts are "real attacks" or "harmless false positives." While it showed certain results, misjudgments increased as time passed, revealing that continuous accuracy checks are necessary to automatically close alerts using AI alone.
Terms Used in This Report True Positive: An alert that was genuinely an attack False Positive: An alert that was actually not a problem Precision: The rate of items judged by the AI as a specific classification that were actually correct Recall: The rate of alerts that the AI successfully caught out of all alerts that should belong to that classification Confidence: How certain the AI is about its own judgment Calibration: Adjusting and checking whether the AI's confidence matches its actual correctness Distribution Shift: Changes in operational alerts compared to training data due to new attacks or product updates
3.
Defense Challenges to Solve SOCs receive a huge volume of alerts, and human review alone takes time The team wants to quickly separate real attacks from false positives It is necessary not only to output classification results, but also to judge which decisions can be processed automatically Accuracy may drop over time or due to environmental differences, even if accuracy is high during training
4.
Proposed Methods and Architecture The research combined an AI that classifies alerts with another AI that checks the certainty of the answers.
The first AI reads the endpoint alert and judges whether it is an attack or a false positive, along with a reason.
The second AI reads the original alert, the judgment, and the reason, and calculates the probability that the answer is correct.
High-confidence results become candidates for automated processing or priority investigation.
Low-confidence results are reviewed by human analysts.
5.
AI Inputs and Outputs Inputs The process that triggered the alert The parent process and its grandparent process Execution command, file name, and file path How often the file is observed within the specific customer environment How often the file is observed across all CrowdStrike customer environments Severity assigned by the sensor, action results, and MITRE ATT&CK classification Explanations left by analysts in the past Whether any input fields are missing Outputs Classification as a real attack or a harmless false positive The reason for the judgment The probability calculated by a separate model that the answer is expected to be correct Information to route the alert to automated processing or human review
6.
AI Processing Flow / System Architecture A security alert occurs on a Windows endpoint.
The alert and related information are formatted as a JSON with multiple fields.
The classification AI reads the information and generates reasons and classification results.
The calibration AI calculates the probability that the classification AI's answer is correct.
High-confidence results become candidates for automated processing or priority investigation.
Low-confidence results are sent to analysts.
The false-positive rate is measured even after operations start, and thresholds are adjusted.
7.
Evaluation Design Data Training: 388,336 items, 8 consecutive weeks of Windows endpoint alerts Tuning: 59,162 items, the following 2 weeks Final Test: 42,686 items, 1 subsequent week Past human judgments of true positives and false positives are used as ground truth.
Evaluation Methods Overall a