:red_circle: HIGH SEV 路 Inference API Producer Sil...
# alarms-prod
a
馃敶 HIGH SEV 路 Inference API Producer Silent (ozone-api stopped sending events) 馃敟 No lock transitions in 3h - ozone-api has likely stopped producing <|馃搱 graph> 路 <|馃搳 service dashboard> 路 馃毃 alarm definition 路 馃敃 silence _HIGH SEV - permanent loss, same class as the proxy rule. ozone-api produces IosFlowBatch + LockState. If it goes quiet, lock state goes stale and sessions stop gating correctly; that time is not backfillable either. THIS IS A PROXY SIGNAL, NOT A DIRECT MEASUREMENT. ozone-api exports no produce metric of its own (the open half of CLE-456), and inference_consumer_records_consumed_total is labelled only by 'topic' - there is no payload-kind label. So this watches lock transitions, which are DERIVED from api-produced LockState: if they stop, the api producer stopped. WHY 3h: across 3 days of prod the minimum in any 3h window was ~18 transitions, so a floor of 5 is ~3.6x margin and never trips on the overnight trough. THE COST IS UP TO 3h DETECTION LATENCY, which is poor against the morning parent-digest deadline (the proxy equivalent catches in 15m). NOTE ON THE QUERY WINDOW: relativeTimeRange is deliberately 600s, NOT 3h. The [3h] range selector already supplies the lookback; setting the window to 3h made Grafana request 11,100 points and the datasource rejected it (max 11,000), which put this rule in a permanent exec-error state. THE REAL FIX is ozone-inference PR #284, which adds a 'kind' label to records_consumed_total - one change, exact per-producer liveness for BOTH producers in minutes. Once merged and deployed, replace this rule with: sum(rate(inference_consumer_records_consumed_total{env="production", kind="ios_flows"}[15m])) < <floor measured post-deploy>._ Grafana vmain-b1e6358