:red_circle: HIGH SEV · API Observation Agent Prod...
# alarms-prod
a
🔴 HIGH SEV · API Observation Agent Produced Nothing 🔥 ozone-api has completed no observation runs in 24 hours <|📈 graph> · <|📊 service dashboard> · 🚨 alarm definition · 🔕 silence _Not one family's observation run has finished in 24 hours. Every family has missed its morning recap — the feature is down, not degraded. Threshold is zero, not a floor, and that is deliberate. Counting completions and alerting below a baseline would be the more sensitive rule, but the log stream is not reliable enough to support a fractional threshold: on 2026-09-03 family 487a35dc completed a real run at 12:31 (confirmed in Langfuse, queued a real recap) and produced no log line at all in either Loki or Render. A floor of 'fewer than 5 of the usual 7' would false-fire the first time two lines are dropped. Zero requires losing every line, which is a different order of unlikely. If you want the sensitive version it has to be built on outreach_messages rows, which cannot be dropped. The 24 hour window is why this is the backstop rather than the fast signal. All eligible runs happen between 09:30 and 13:30 UTC (5:30 and 6:30 local across America/New_York through America/Los_Angeles — verified, no families outside North American timezones), so a trailing window short enough to fire the same morning would read zero all afternoon and page every night. 'API Observation Agent Cron Not Firing' catches a dead scheduler within 90 minutes and 'API Observation Agent Run Failures' catches failing runs within the hour; this one exists to catch the case where both are silent and nothing happened anyway. To get same-morning notification on this condition, the rule needs a 5h window plus a notification time interval restricting it to roughly 1400 1500 UTC. Worth doing if a whole-day miss ever actually happens. noDataState and execErrState are Alerting: absence is the condition._ Grafana vmain-b1e6358