🔴 HIGH SEV · Mock-IP Allocation Failure Rate Critical (DNS broken for a real share of lookups)
🔥 Over 5% of DNS lookups in the tunnel are failing — kids' apps are intermittently not loading
<|📈 graph> · <|📊 service dashboard> ·
🚨 alarm definition ·
🔕 silence
_IMPACT: roughly 1 in 20 first-time domain lookups returns SERVFAIL, so apps fail to load. Expect support contact. It gets WORSE the longer it runs, because already-visited sites keep working from the device's 60s DNS cache and only new lookups break.
NOT AT RISK: screen-time data. DNS failures are retryable; nothing is permanently lost and no enforcement is bypassed.
CAUSE: ozone-proxy cannot allocate mock IPs because Redis calls are failing (2s timeout). There is no resolver fallback — all tunnel DNS is forced to our own mock server, so the device has nowhere else to go.
DO:
1. Is Redis up and responsive? Check 'Redis Down', 'Redis Command Latency High', 'Redis Fork Stall'.
2. Is the pool exhausted rather than erroring? Check 'Mock-IP Pool Exhausted' — different fix.
3. If Redis looks healthy, suspect proxy→Redis connectivity or a maxclients ceiling ('Redis Connections Rejected').
NORMAL: ~1 error per week out of ~1.8M lookups. Escalated from the warning tier 'Mock-IP Allocation Errors Elevated'.
Threshold is a RATIO with a 20-error floor on purpose — do not replace it with a count. A count-based version paged Urgent for a single transient error (CLE-699)._
Grafana vmain-b1e6358