:red_circle: HIGH SEV 路 Inference In-Log Wait Seve...
# alarms-prod
a
馃敶 HIGH SEV 路 Inference In-Log Wait Severe (p99 over 60s) 馃敟 In-log wait p99 over 60s for 5m - enforcement is materially behind <|馃搱 graph> 路 <|馃搳 service dashboard> 路 馃毃 alarm definition 路 馃敃 silence _HIGH SEV escalation of the LOW SEV 'Inference Consume Latency p99 High' (3s). THRESHOLD 60s / for 5m (changed 2026-09-25 from 30s / 15m). This page is about ENFORCEMENT, not notifications. Lost block notifications are low-sev (they start dropping at ~22s of lag and are covered by the warning + 'Block-Notify SLO Miss'). In-log wait measured in MINUTES is what is critical: screen time is being billed minutes late, so limits trip and blocks land minutes late. 5m because a sustained minute-plus backlog is already user-visible; the old 15m let the 09-07 stall (peak 29.45s) and the start of the 09-09 escalation (peak >=300s) run unpaged. HOW THIS REACHES ENFORCEMENT (indirect but decisive): the proxy's block decision is synchronous and domain-driven - resolve domain -> activityIds, check the proxyblock{device}:{activity} Redis key - and never consults a foreground prediction at connection time. What predictions govern is whether that key is ever WRITTEN: inference attributes screen time, the limit is reached, ozone-api sets the key. So a consumer running behind delays the prediction, the billed time, the threshold, and finally the block - and the kid keeps using the app past their limit meanwhile. Past ~600s (STORAGE_WINDOW_MS) event time is DROPPED (catch_up_overflow) - real screen-time loss. BUCKETS: histogram boundaries are ...10, 30, 60, 120, 300, +Inf. 60 sits on a boundary. 300 is the top finite bucket, so p99 CANNOT read above 300 - a pegged 300 means >=5min, unbounded. Prod baseline 0.05-0.48s hourly p99 over the 14d to 09-25 (after the 3rd consumer box). Should be silent in normal operation. FIRST CHECK: /d/ozone-inference row 170 'Consumer Stall Diagnosis' - panel 171 poll-vs-process, 172 in-log wait per worker, 177 replay depth. Known mechanism = poll starvation by expensive cadences at evening peak (CLE-425). no_data=OK: histogram_quantile returns NaN when the consumer is gone, and that case is covered by the coverage/starved criticals._ Grafana vmain-2312ebe
馃憖 1