🔴 HIGH SEV · API Observation Agent Produced Nothing
🔥 ozone-api has completed no observation runs in 24 hours
<|📈 graph> · <|📊 service dashboard> ·
🚨 alarm definition ·
🔕 silence
_Not one family's observation run has finished in 24 hours. Every family has missed its morning recap — the feature is down, not degraded.
Threshold is zero, not a floor, and that is deliberate. Counting completions and alerting below a baseline would be the more sensitive rule, but the log stream is not reliable enough to support a fractional threshold: on 2026-09-03 family 487a35dc completed a real run at 12:31 (confirmed in Langfuse, queued a real recap) and produced no log line at all in either Loki or Render. A floor of 'fewer than 5 of the usual 7' would false-fire the first time two lines are dropped. Zero requires losing every line, which is a different order of unlikely. If you want the sensitive version it has to be built on outreach_messages rows, which cannot be dropped.
The 24 hour window is why this is the backstop rather than the fast signal. All eligible runs happen between 09:30 and 13:30 UTC (5:30 and 6:30 local across America/New_York through America/Los_Angeles — verified, no families outside North American timezones), so a trailing window short enough to fire the same morning would read zero all afternoon and page every night. 'API Observation Agent Cron Not Firing' catches a dead scheduler within 90 minutes and 'API Observation Agent Run Failures' catches failing runs within the hour; this one exists to catch the case where both are silent and nothing happened anyway.
To get same-morning notification on this condition, the rule needs a 5h window plus a notification time interval restricting it to roughly 14
00 1500 UTC. Worth doing if a whole-day miss ever actually happens.
noDataState and execErrState are Alerting: absence is the condition._
Grafana vmain-b1e6358