StepShield: When, Not Whether to Intervene on Rogue Agents

Agent safety benchmarks measure whether a monitor detects harm, not when.

Yet timing is the difference between intervention and autopsy.

We introduce StepShield, the first benchmark that treats detection timeliness as a first-class metric.

On 9,429 incident-grounded code-agent trajectories, we define the Early Intervention Rate (EIR): the fraction of detected rogue trajectories where the alert fires within a k-step window after the divergence point, isolating timing quality from coverage.

This metric exposes what we call the Forensics Trap: a pattern-based guardrail with 847 rules achieves 86% recall yet is statistically indistinguishable from random timing on EIR (0.23 vs.

0.24; p = 0.66, one-sided binomial; difference within CI), because over three-quarters of its alerts trigger on benign prefix code before any violation occurs.

The 4x EIR gap between rule-based and semantic detectors is completely invisible to accuracy, recall, or F1.

Our finding is structural: regex guardrails detect syntax, not intent, and therefore cannot distinguish the moment an agent turns rogue, rendering the entire deployed class of pattern-based monitors unsuited for real-time oversight.

No existing method simultaneously achieves high recall, low false-positive rate, and timely intervention, establishing step-level rogue detection as genuinely unsolved.