AlarmBot
09/25/2026, 9:04 PMsuppressed (31 on prod, vs 87 sent). This rule counts ONLY notifies we meant to send and dropped.
WHAT live ACTUALLY MEANS โ READ THIS BEFORE BLAMING A PHONE
live is now - GLOBAL frontier <= LIVE_THRESHOLD_MS (7s), evaluated AT THE INSTANT THE TIMER FIRES. The frontier is the max event time across ALL devices, so it tracks the FASTEST device, and a catch-up drain that has just finished reads as live. So this metric does NOT prove we were healthy the whole time โ only that we were caught up in the moment we decided to give up.
FIRST THING TO CHECK: WAS THE CONSUMER BEHIND THE LOG?
inference_consumer_consume_latency_seconds (in-log wait) for that instance at that minute is the ONLY metric that isolates consumer lateness from phone-upload lateness. p99 well above the usual 0.1-0.5s means WE were behind, and the notify was late because the wheel could not fire: the frontier freezes while the consumer is not pulling, the fire gate stays shut, then a drain advances every cascade tier at once and the send is judged against a wall clock already past budget.
DO NOT use max_device_lag to exonerate the consumer. Consumer lag inflates device lag, so a high max_device_lag is consistent with EITHER cause and proves neither.
THEN get the device and app from Loki:
{env="production"} |= "block-notify SLO miss"
The line carries device_id, alias and the age in ms.
ESCALATE IF: in-log wait was elevated (the consumer-stall case โ this is a real defect, not a slow phone); several misses in a day; or misses clustering on one device or app.
WHY THIS METRIC AND NOT THE LATENCY HISTOGRAM: the histogram also observes held catch-up drains whose age is minutes-to-hours, so its p99 reads ~238s on prod and is not an SLO signal.
BASELINE (re-measured 2026-09-08): 87 sent / 31 suppressed / 1 SLO miss, ~40 sends/day. Counters RESET ON DEPLOY, so these are since-restart, not lifetime. One miss every week or two is today's normal.
MISSES SO FAR, both consumer stalls on inference-consumer-prod-us-west-2, not slow phones:
2026-09-07 13:09Z โ age 53.8s, Snapchat, device 314e32e1. Confirmed foreground: modeled bundle (AUROC 0.996) whose cascade tier CLEARED, session billed 22.5s unsuppressed, predictor score 1778 vs runner-up 8, phone unlocked across the block. The block_event bound 1.6s after the block, so the pipeline was current at the block; the loss came from ~42s of dead time after the last cascade tier came due, while in-log wait p99 on that box spiked to 29-51s.
2026-09-08 002851Z โ age 869.8s (14.5 min), device 6a63ddb3, same box, during a stall that pegged in-log wait p99 at the 300s top bucket for ~20 minutes.
Root cause of the stalling itself is still not established._
Grafana vmain-bff6a80