🟠 LOW SEV · Inference Block-Notify SLO Miss 🔥 2+ block notifications dropped within an hour — those kids were blocked with no explanation <|📈 graph> · <|📊 service dashboard> · :rotating_light: alarm definition · :no_bell: silence _WARNING. THIS IS THE NOTIFICATION, NOT THE BLOCK. The traffic was still cut: ozone-proxy blocks inline off proxyblock{device}:{activity} in Redis and never consults inference. No screen time is miscounted and no limit went unenforced. What was lost is the EXPLANATION — one kid's app went dead with no on-device notice. THRESHOLD: 2+ IN ONE HOUR, DELIBERATELY NOT 1 (changed 2026-09-08) A lone miss is expected background at this volume (~40 sends/day; 1 miss in the 14d before this change) and is usually one phone or one short stall — it is NOT silence-worthy harm, so it no longer alerts. What this now catches is a CLUSTER: several explanations lost inside the same hour, which is the shape of a real regression rather than noise. Isolated misses are still recorded and still visible on /d/ozone-inference row 'Block-Notify Gate' (SLO-miss rate + total panels) and in Loki — check there before assuming a quiet week means zero. NOTE: the 09-07 incident described below cost exactly ONE notify and would NOT fire under this threshold. That is intended: its consumer stall is the in-log-wait alerts' job to catch. WHAT HAPPENED A block notify reached its send decision more than 22s after the block's event time (BLOCK_NOTIFY_FRESHNESS_MS 20s + HYSTERESIS 2s). Past that budget the code HOLDS, terminally: it returns before the api.session_blocked call, writes a durable decision marker, and never retries. Each count = one push lost forever, NOT a late push. NOT THE SAME AS A DELIBERATE NON-NOTIFY. We intentionally stay silent when the ML cascade judges the blocked activity to be background — that path counts as

suppressed

(31 on prod, vs 87 sent). This rule counts ONLY notifies we meant to send and dropped. WHAT

live

ACTUALLY MEANS — READ THIS BEFORE BLAMING A PHONE

live

is

now - GLOBAL frontier <= LIVE_THRESHOLD_MS

(7s), evaluated AT THE INSTANT THE TIMER FIRES. The frontier is the max event time across ALL devices, so it tracks the FASTEST device, and a catch-up drain that has just finished reads as live. So this metric does NOT prove we were healthy the whole time — only that we were caught up in the moment we decided to give up. FIRST THING TO CHECK: WAS THE CONSUMER BEHIND THE LOG?

inference_consumer_consume_latency_seconds

(in-log wait) for that instance at that minute is the ONLY metric that isolates consumer lateness from phone-upload lateness. p99 well above the usual 0.1-0.5s means WE were behind, and the notify was late because the wheel could not fire: the frontier freezes while the consumer is not pulling, the fire gate stays shut, then a drain advances every cascade tier at once and the send is judged against a wall clock already past budget. DO NOT use

max_device_lag

to exonerate the consumer. Consumer lag inflates device lag, so a high max_device_lag is consistent with EITHER cause and proves neither. THEN get the device and app from Loki: {env="production"} |= "block-notify SLO miss" The line carries device_id, alias and the age in ms. ESCALATE IF: in-log wait was elevated (the consumer-stall case — this is a real defect, not a slow phone); several misses in a day; or misses clustering on one device or app. WHY THIS METRIC AND NOT THE LATENCY HISTOGRAM: the histogram also observes held catch-up drains whose age is minutes-to-hours, so its p99 reads ~238s on prod and is not an SLO signal. BASELINE (re-measured 2026-09-08): 87 sent / 31 suppressed / 1 SLO miss, ~40 sends/day. Counters RESET ON DEPLOY, so these are since-restart, not lifetime. One miss every week or two is today's normal. THE ONLY MISS SO FAR (2026-09-07 13:09Z, age 53.8s, Snapchat, device 314e32e1) WAS A CONSUMER STALL, NOT A SLOW PHONE. Confirmed foreground: modeled bundle (AUROC 0.996) whose cascade tier CLEARED, session billed 22.5s unsuppressed, predictor score 1778 vs runner-up 8, phone unlocked across the block. The block_event itself bound 1.6s after the block, so the pipeline was current at the block; the loss came from ~42s of dead time after the last cascade tier came due, while in-log wait p99 on that box was spiking to 29-51s and cadences were burst-draining. Root cause of the stalling itself was never established._ Grafana vmain-434316c