🟠 LOW SEV (2 servers) · API Observation Agent Exhausted Both Attempts 🔥 ozone-api observation run failed for family 240663cd-bbb4-4b47-9a1f-7155abfe34fd with no successful retry <|📈 graph> · <|📊 service dashboard> · :rotating_light: alarm definition · :no_bell: silence 🔥 ozone-api observation run failed for family 5835c49c-3fb2-425d-8204-b054ecd68a29 with no successful retry <|📈 graph> · <|📊 service dashboard> · :rotating_light: alarm definition · :no_bell: silence _A family's daily observation run failed and never succeeded, so that family gets no morning recap. Fires per family_id. Deliberately NOT an alert on any failure. Two layers of retry can save a run, and neither deserves a page: 1. In-request: a flex 429 is retried immediately on standard processing (openAI.flexCapacityFallback). If that works the run completes and nothing is logged as a failure at all, so it never reaches this rule. 2. Scheduled: each family gets two attempts, 5:30 and 6:30 local. On 2026-09-03 two of ten families failed the first and succeeded the second. The 'unless' clause encodes that: it returns failure series only for families with no runObservationAgent.complete in the same window. Verified as an ad-hoc instant query against 09-03: exactly the 8 families that lost the day, excluding 4b5d675e and b653e296, which recovered on the 6:30 retry. TWO QUERY SETTINGS ARE LOAD-BEARING, both learned the hard way while building this rule: queryType 'instant'. The window must be one point at eval time, which is what the 'unless' logic assumes. relativeTimeRange.from kept SMALL (60s), with the 5h window expressed only by the [5h] range selector. Grafana appears to evaluate the instant query at (now - relativeTimeRange.from), so setting from=18000 to 'mean' 5h silently shifted the whole window back five hours: the rule went Pending on b653e296, whose 12:31 failure was in the shifted window while its 13:31 recovery fell outside it. Do not set from to the window length. The 65m 'for' is load-bearing too. Between the 5:30 failure and the 6:30 attempt the condition is transiently true for every family, so a shorter 'for' would fire on runs about to recover. 65m outlasts that gap: a Pacific family that fails both attempts fires at ~13:36 UTC, about five minutes after its last chance. Known false positive: a family that fails once, recovers, and has its complete line dropped from the log stream. Line loss is real — see 'API Observation Agent Produced Nothing' — but it has to coincide with a failure for the same family, which is much rarer. Baseline is zero: none at all 2026-08-28 to 09-02, then 8 lost families on 09-03 when OpenAI flex capacity ran out._ Grafana vmain-ed3d476