{"stable_id":"7405cae4f09ed2d4","slug":"azure-warden","company":"Microsoft Azure","workflow_name":"Warden automated incident detection in IcM","function_code":"it_operations","pattern_codes":["coordination_compression","continuous_decisioning"],"changed_assumption":"A cross-service Azure incident is recognized when a human on-call, seeing only a partial service view, pieces together cascading alerts and declares the incident.","evidence_strength":"verified","publication_tier":"showcase","freshness":"current","reviewed_at":"2026-08-23","updated_at":"2026-08-22","source_quality_summary":"1 peer reviewed, 1 primary; publication outcomes are verified.","caveat_summary":"The public paper does not report an absolute minute delta; time is in normalized time units. The 68% figure is among successfully detected incidents, not all incidents. Comparison is detection latency versus historical human declaration for the same incident, not operating cost or full mitigation time. No public 2026 confirmation that Warden remains the production detector under that name.","freshness_caveat":null,"workflow_summary":{"before":[{"actor":"Service monitors","action":"Fire alerts inside individual Azure services as local symptoms appear.","actor_type":"control"},{"actor":"On-call engineers","action":"Triage service alerts and collaborate to recognize one cross-service incident.","actor_type":"control"},{"actor":"Incident Commander","action":"Coordinates mitigation after the incident is declared.","actor_type":"control"}],"hinge":"A cross-service Azure incident is recognized when a human on-call, seeing only a partial service view.","after":[{"actor":"Service monitors","action":"Continue to fire alerts; Warden ingests them globally from IcM.","actor_type":"system"},{"actor":"Warden detector","action":"Every minute, scores recent alerts and flags potential incidents at a greater-than-90% precision operating point.","actor_type":"ai"},{"actor":"On-call engineers","action":"Receive a Recommended Actions notification with grouped incident-indicating alerts and an Emerging Issues dashboard.","actor_type":"system"}],"decision_mode":"moved","decision_marker":"Selection moves from a fixed rule to the model"},"before":[{"order":1,"actor":"Service monitors","action":"Fire alerts inside individual Azure services as local symptoms appear.","handoff_to":"Owning service on-call","control":"Per-service monitors; each team has only a partial view of the platform"},{"order":2,"actor":"On-call engineers","action":"Triage service alerts and collaborate to recognize one cross-service incident.","handoff_to":"Incident Commander","control":"Human pattern recognition across teams; motivating example took nearly 50 minutes to declare"},{"order":3,"actor":"Incident Commander","action":"Coordinates mitigation after the incident is declared.","handoff_to":"Mitigation owners","control":"IcM incident declaration"}],"after":[{"order":1,"actor":"Service monitors","action":"Continue to fire alerts; Warden ingests them globally from IcM.","handoff_to":"Warden","control":"Selected monitors with high weighted mutual information versus historical incidents"},{"order":2,"actor":"Warden detector","action":"Every minute, scores recent alerts and flags potential incidents at a greater-than-90% precision operating point.","handoff_to":"On-call engineers","control":"Precision held above 90% (recall ~58%, F1 0.71); late detections relative to humans are not notified"},{"order":3,"actor":"On-call engineers","action":"Receive a Recommended Actions notification with grouped incident-indicating alerts and an Emerging Issues dashboard, then prioritize and start cross-team collaboration.","handoff_to":"Incident Commander / mitigation owners","control":"Humans still declare and mitigate; Warden does not auto-close or auto-mitigate the incident"}],"decision_rights":"Warden decides when to notify and which alerts to group. Humans retain incident declaration, severity, and mitigation. Notifications are suppressed when Warden would be slower than the human declaration already in progress.","exception_path":"False positives are constrained by a >90% precision threshold. Incidents Warden misses wait for human declaration. Not all incidents are covered by monitors; the authors flag uncovered incidents as future work.","removed_work":["Ad-hoc multi-team discussion as the primary way to recognize a cross-service incident","Each on-call independently deciding whether a local alert is isolated or part of a larger event"],"outcomes":[{"metric":"Share of successfully detected incidents where Warden is faster than human declaration","baseline":"On-call incident declaration time for the same incidents","result":"Warden faster in ~68% of successfully detected incidents; median saving 21.8 NTUs, about 15% of time-to-mitigate for those cases","period":"18 months of Azure IcM data starting October 2018 (last 2 months held out for test); production deployment about 3 months through May 2020","scale":"26 major Azure services accounting for ~72% of Azure incidents; >10 million alerts; ~240 GB","attribution_caveat":"The public paper does not report an absolute minute delta; time is in normalized time units. The 68% figure is among successfully detected incidents, not all incidents. Comparison is detection latency versus historical human declaration for the same incident, not operating cost or full mitigation time. No public 2026 confirmation that Warden remains the production detector under that name.","evidence_label":"verified"}],"executive_lesson":"The scarce job in a large cloud is not watching one service's alerts. It is noticing that several services are the same incident. Azure moved that recognition from on-call discussion to a global detector that pages the right people, and was faster than human declaration in about two-thirds of the incidents it caught.","anti_pattern":"Reporting '68% faster than human' as a minute-scale MTTR cut, or implying Warden closed incidents without an on-call.","questions_for_leaders":["Is the outcome time-to-detect versus the same incident's human declaration, or a modeled counterfactual?","What precision floor is high enough that on-calls will trust the page?","If the detector is late, do we suppress the notification, or create a second incident?"],"collections":["queue-eliminated","embodied-work"],"bundle_version":"1.0.0","bundle_fingerprint":"sha256:c23c6cc2b88153d008ea8fda928f632ce0011fc2d4c5036672a16e5d895bab93","canonical_url":"https://brianletort.ai/transformations/azure-warden"}