:large_orange_circle: LOW SEV ยท Inference Block-No...
# alarms-prod
a
๐ŸŸ  LOW SEV ยท Inference Block-Notify SLO Miss ๐Ÿ”ฅ 2+ block notifications dropped within an hour โ€” those kids were blocked with no explanation <|๐Ÿ“ˆ graph> ยท <|๐Ÿ“Š service dashboard> ยท :rotating_light: alarm definition ยท :no_bell: silence _WARNING. THIS IS THE NOTIFICATION, NOT THE BLOCK. The traffic was still cut: ozone-proxy blocks inline off proxyblock{device}:{activity} in Redis and never consults inference. No screen time is miscounted and no limit went unenforced. What was lost is the EXPLANATION โ€” one kid's app went dead with no on-device notice. THRESHOLD: 2+ IN ONE HOUR, DELIBERATELY NOT 1 (changed 2026-09-08) A lone miss is expected background at this volume (~40 sends/day; 1 miss in the 14d before this change) and is usually one phone or one short stall โ€” it is NOT silence-worthy harm, so it no longer alerts. What this now catches is a CLUSTER: several explanations lost inside the same hour, which is the shape of a real regression rather than noise. Isolated misses are still recorded and still visible on /d/ozone-inference row 'Block-Notify Gate' (SLO-miss rate + total panels) and in Loki โ€” check there before assuming a quiet week means zero. NOTE: the 09-07 incident described below cost exactly ONE notify and would NOT fire under this threshold. That is intended: its consumer stall is the in-log-wait alerts' job to catch. THE EVALUATOR IS 'gt 1.5', NOT 'gt 2' โ€” AND NOT 'gt 1'. DO NOT 'CORRECT' IT. increase() EXTRAPOLATES over the window: a SINGLE miss evaluates to ~1.0045, not 1.0. The first attempt at this threshold used 'gt 1' and therefore still fired on one miss โ€” it did exactly that at 00:34Z on 2026-09-09, during the us-west-2 stall, which is how the bug was found. Any integer-count threshold sitting on top of increase() needs the .5 offset to mean what it says. WHAT HAPPENED A block notify reached its send decision more than 22s after the block's event time (BLOCK_NOTIFY_FRESHNESS_MS 20s + HYSTERESIS 2s). Past that budget the code HOLDS, terminally: it returns before the api.session_blocked call, writes a durable decision marker, and never retries. Each count = one push lost forever, NOT a late push. NOT THE SAME AS A DELIBERATE NON-NOTIFY. We intentionally stay silent when the ML cascade judges the blocked activity to be background โ€” that path counts as
suppressed
(31 on prod, vs 87 sent). This rule counts ONLY notifies we meant to send and dropped. WHAT
live
ACTUALLY MEANS โ€” READ THIS BEFORE BLAMING A PHONE
live
is
now - GLOBAL frontier <= LIVE_THRESHOLD_MS
(7s), evaluated AT THE INSTANT THE TIMER FIRES. The frontier is the max event time across ALL devices, so it tracks the FASTEST device, and a catch-up drain that has just finished reads as live. So this metric does NOT prove we were healthy the whole time โ€” only that we were caught up in the moment we decided to give up. FIRST THING TO CHECK: WAS THE CONSUMER BEHIND THE LOG?
inference_consumer_consume_latency_seconds
(in-log wait) for that instance at that minute is the ONLY metric that isolates consumer lateness from phone-upload lateness. p99 well above the usual 0.1-0.5s means WE were behind, and the notify was late because the wheel could not fire: the frontier freezes while the consumer is not pulling, the fire gate stays shut, then a drain advances every cascade tier at once and the send is judged against a wall clock already past budget. DO NOT use
max_device_lag
to exonerate the consumer. Consumer lag inflates device lag, so a high max_device_lag is consistent with EITHER cause and proves neither. THEN get the device and app from Loki: {env="production"} |= "block-notify SLO miss" The line carries device_id, alias and the age in ms. ESCALATE IF: in-log wait was elevated (the consumer-stall case โ€” this is a real defect, not a slow phone); several misses in a day; or misses clustering on one device or app. WHY THIS METRIC AND NOT THE LATENCY HISTOGRAM: the histogram also observes held catch-up drains whose age is minutes-to-hours, so its p99 reads ~238s on prod and is not an SLO signal. BASELINE (re-measured 2026-09-08): 87 sent / 31 suppressed / 1 SLO miss, ~40 sends/day. Counters RESET ON DEPLOY, so these are since-restart, not lifetime. One miss every week or two is today's normal. MISSES SO FAR, both consumer stalls on inference-consumer-prod-us-west-2, not slow phones: 2026-09-07 13:09Z โ€” age 53.8s, Snapchat, device 314e32e1. Confirmed foreground: modeled bundle (AUROC 0.996) whose cascade tier CLEARED, session billed 22.5s unsuppressed, predictor score 1778 vs runner-up 8, phone unlocked across the block. The block_event bound 1.6s after the block, so the pipeline was current at the block; the loss came from ~42s of dead time after the last cascade tier came due, while in-log wait p99 on that box spiked to 29-51s. 2026-09-08 002851Z โ€” age 869.8s (14.5 min), device 6a63ddb3, same box, during a stall that pegged in-log wait p99 at the 300s top bucket for ~20 minutes. Root cause of the stalling itself is still not established._ Grafana vmain-bff6a80