https://linen.dev logo
Join Slack
Powered by
# alarms-prod
  • a
    🚨 Panic switch activated Family: Smith Family (
    2cfc02ce-5293-41bd-b4f8-018bdcd6effc
    ) Triggered by: Marisa Smith (
    uL6BWIDrFnO7gMwCvww1KjzkuZu2
    ) Target device: Amelia’s iPhone (
    f3596b2d-9be6-46b7-b038-2c516251b15a
    ) Clearspace controls (content filter + VPN) are now fully disabled on this device until re-enabled.
  • a
    ✅ Protections resumed Family: Smith Family (
    2cfc02ce-5293-41bd-b4f8-018bdcd6effc
    ) Triggered by: Marisa Smith (
    uL6BWIDrFnO7gMwCvww1KjzkuZu2
    ) Target device: Amelia’s iPhone (
    f3596b2d-9be6-46b7-b038-2c516251b15a
    ) Clearspace controls (content filter + VPN) are being turned back on for this device.
  • a
    ✅ Protections resumed Family: Topacio Family (
    2bd44ba3-79ca-4603-bda2-a2c793e1dc81
    ) Triggered by: Pamela C. Topacio (
    ESA7xvcl1WTQi0WqRpaSrrxLw6n1
    ) Target device: Atlas’s iPhone (
    6743c544-f62f-40d6-abd7-4c571e1a3c57
    ) Clearspace controls (content filter + VPN) are being turned back on for this device.
  • a
    ✅ Protections resumed Family: Topacio Family (
    2bd44ba3-79ca-4603-bda2-a2c793e1dc81
    ) Triggered by: Pamela C. Topacio (
    ESA7xvcl1WTQi0WqRpaSrrxLw6n1
    ) Target device: Eames’s iPhone (
    da9c6329-819f-417c-9d19-2ecca2d8d7ba
    ) Clearspace controls (content filter + VPN) are being turned back on for this device.
  • a
    🚨 Panic switch activated Family: DiPasquale Family (
    693027eb-d6d8-4aa5-a048-abe5a8e450ed
    ) Triggered by: Andrea DiPasquale (
    Zcih3ch6XSMoENwpMQ83VsB1WSv1
    ) Target device: Naz’s iPad (
    b33e3d90-118d-4e8f-863d-67810aa82e28
    ) Clearspace controls (content filter + VPN) are now fully disabled on this device until re-enabled.
  • a
    🚨 Panic switch activated Family: Topacio Family (
    2bd44ba3-79ca-4603-bda2-a2c793e1dc81
    ) Triggered by: Pamela C. Topacio (
    ESA7xvcl1WTQi0WqRpaSrrxLw6n1
    ) Target device: Atlas’s iPhone (
    6743c544-f62f-40d6-abd7-4c571e1a3c57
    ) Clearspace controls (content filter + VPN) are now fully disabled on this device until re-enabled.
  • a
    🚨 Panic switch activated Family: Topacio Family (
    2bd44ba3-79ca-4603-bda2-a2c793e1dc81
    ) Triggered by: Pamela C. Topacio (
    ESA7xvcl1WTQi0WqRpaSrrxLw6n1
    ) Target device: Eames’s iPhone (
    da9c6329-819f-417c-9d19-2ecca2d8d7ba
    ) Clearspace controls (content filter + VPN) are now fully disabled on this device until re-enabled.
  • a
    🟠 LOW SEV · Ozone Proxy DNS Failures Elevated 🔥 DNS resolution failures elevated on vpn-prod-us-west-1 :chart_with_upwards_trend: graph · :bar_chart: service dashboard · :rotating_light: alarm definition · :no_bell: silence _Domains are failing to resolve, so clients get nothing back. Usually systemd-resolved or upstream DNS when it spikes; a low baseline is normal because users visit dead domains. Threshold 0.2/s against a measured 24h maximum of 0.052/s at this window, so roughly 4x headroom. These are bursty — an instantaneous read is usually 0, so do not treat a quiet moment as the baseline. Check Loki for '[PROXY] DNS lookup failed' and note which domains are affected. NXDOMAIN on dead ad domains is logged at WARN and is expected; resolver-down errors are logged at ERROR._ Grafana vmain-458abfb
  • a
    ✅ RESOLVED · Ozone Proxy DNS Failures Elevated ✅ resolved: DNS resolution failures elevated on vpn-prod-us-west-1 Grafana vmain-458abfb
  • a
    ✅ Protections resumed Family: Topacio Family (
    2bd44ba3-79ca-4603-bda2-a2c793e1dc81
    ) Triggered by: Pamela C. Topacio (
    ESA7xvcl1WTQi0WqRpaSrrxLw6n1
    ) Target device: Eames’s iPhone (
    da9c6329-819f-417c-9d19-2ecca2d8d7ba
    ) Clearspace controls (content filter + VPN) are being turned back on for this device.
  • a
    ✅ Protections resumed Family: Topacio Family (
    2bd44ba3-79ca-4603-bda2-a2c793e1dc81
    ) Triggered by: Pamela C. Topacio (
    ESA7xvcl1WTQi0WqRpaSrrxLw6n1
    ) Target device: Atlas’s iPhone (
    6743c544-f62f-40d6-abd7-4c571e1a3c57
    ) Clearspace controls (content filter + VPN) are being turned back on for this device.
  • a
    🚨 Panic switch activated Family: Hoxie Family (
    eaa8a492-17f3-4c05-b7b5-4a3d1b1d1839
    ) Triggered by: Anne-Marie Hoxie (
    furoPz6VNENJBumKNiNmqDh3eur1
    ) Target device: Addy’s iPad (
    e0e7f82c-2af9-4b26-bec8-3c9f95f64d97
    ) Clearspace controls (content filter + VPN) are now fully disabled on this device until re-enabled.
  • a
    🟠 LOW SEV · Ozone Proxy Overhead p99 High 🔥 ozone-proxy is adding over 2.5s of setup latency (p99) :chart_with_upwards_trend: graph · :bar_chart: service dashboard · :rotating_light: alarm definition · :no_bell: silence _Time from accepting a client connection to starting the relay — DNS lookup, enforcement check and upstream dial. This is what a user feels as 'the VPN makes everything slow'. Typical p99 is 0.39s, but the 24h peak at this window is 1.42s, so the spread is wide. Threshold is 2.5s deliberately: an earlier draft used 1.5s, which the measured maximum would have tripped. To localise it, compare the DNS and dial p99s on the dashboard — for direct connections setup should be dominated by dial time. Then query Tempo for connection.setup_duration_ms > 1000 and inspect which child span (proxy.dns or proxy.dial) is long. Rising latency alongside a spike in active tunnels usually means resource pressure on the box rather than a network problem._ Grafana vmain-458abfb
  • a
    ✅ RESOLVED · Ozone Proxy Overhead p99 High ✅ resolved: ozone-proxy is adding over 2.5s of setup latency (p99) Grafana vmain-458abfb
  • m
    FYI just saw this and could be relevant status.hetzner.com/incident/…
    ‼️ 1
  • r
    🚨 Linq Platform - Incident update Webhook Delivery Degradation Status: Investigating Webhook notifications delayed for a subset of phone lines. Messages continue to send and arrive, but some webhooks for events are impacted. ⚠️ Affected components: Linq V3 API
  • r
    🚨 Linq Platform - New incident Webhook Delivery Degradation Status: Resolved From 23:38 UTC on October 31, 2026 to 02:25 UTC on November 1, 2026, webhook notifications were delayed and in some cases not delivered for a subset of phone lines. Messages continued to send and arrive during this time, but some webhooks for events in this window may never have reached the webhook endpoint. The issue is resolved and webhook delivery is back to normal. Please reach out to us if you have any questions.
  • a
    🟠 LOW SEV · Inference Unlocks Without Sessions 🔥 Phones are unlocking but no foreground sessions are starting <|📈 graph> · <|📊 service dashboard> · :rotating_light: alarm definition · :no_bell: silence _LOW SEV, and treat as PROVISIONAL. Catches 'consuming fine but predicting nothing' - no transport metric covers it. Screen time flatlines at zero while every other signal looks healthy, so a parent opens the app and sees 0 minutes for a kid who was on their phone all day. That is corrosive because the product looks like it is working. Rated LOW because this is a HEURISTIC, not a measurement: it assumes unlocks and session-starts stay correlated. That held across 3 days of prod (sessions run ~5-6x unlocks, both troughing to zero together overnight), but a legitimately idle-but-unlocked fleet would trip it. Watch for false positives before promoting. Unlocks are the denominator on purpose - a phone that unlocks should produce a session, at any hour. Overnight both go to zero together and this correctly stays silent: nothing is happening, so nothing is being lost. This replaced a 6h absolute-count window, which could not fire before the morning parent digest. NOTE: restarts and partition moves re-report every device's lock state once as a 'first observation', counting as a transition - so >= 5 is an arming floor, not an exact count._ Grafana vmain-98b4777
  • a
    ✅ RESOLVED · Inference Unlocks Without Sessions ✅ resolved: Phones are unlocking but no foreground sessions are starting Grafana vmain-98b4777
  • a
    🟠 LOW SEV · Ozone Proxy DNS Failures Elevated 🔥 DNS resolution failures elevated on vpn-prod-us-west-1 :chart_with_upwards_trend: graph · :bar_chart: service dashboard · :rotating_light: alarm definition · :no_bell: silence _Domains are failing to resolve, so clients get nothing back. Usually systemd-resolved or upstream DNS when it spikes; a low baseline is normal because users visit dead domains. Threshold 0.2/s against a measured 24h maximum of 0.052/s at this window, so roughly 4x headroom. These are bursty — an instantaneous read is usually 0, so do not treat a quiet moment as the baseline. Check Loki for '[PROXY] DNS lookup failed' and note which domains are affected. NXDOMAIN on dead ad domains is logged at WARN and is expected; resolver-down errors are logged at ERROR._ Grafana vmain-98b4777
  • a
    ✅ RESOLVED · Ozone Proxy DNS Failures Elevated ✅ resolved: DNS resolution failures elevated on vpn-prod-us-west-1 Grafana vmain-98b4777
  • a
    🚨 Panic switch activated Family: Wyk Family (
    fd4f4386-6017-4610-9416-a1d774d503a1
    ) Triggered by: Naomi Ruth Van Wyk (
    C0IR6DWbHeZfeZIw2rsIhR0nczk1
    ) Target device: Esther’s iPad (
    547ba6df-5345-47fe-84d7-5674e691b972
    ) Clearspace controls (content filter + VPN) are now fully disabled on this device until re-enabled.
  • a
    🟠 LOW SEV · API SMS Fallback Sent 🔥 ozone-api sent a parent the SMS fallback error instead of an answer <|📈 graph> · <|📊 service dashboard> · :rotating_light: alarm definition · :no_bell: silence _A parent texted the agent and the turn failed, so sendFallbackSms in runAgentSmsTurn.ts sent them the canned SMS_FALLBACK_ERROR text instead of a reply. The parent saw the failure, which is why this fires on any single occurrence (gt 0) with no 'for' — the fallback is the last resort after every retry, so one is already a bad experience worth looking at. Each line carries session_id plus the session log attributes. Look up the preceding error for that session in Loki to find the actual cause (LLM provider error, DB unreachable, Linq send failure). smsFallbackSendFailed is the sibling event when even the fallback text could not be delivered — it is not covered by this rule. Baseline is zero: the event was added on 2026-09-22 and had never been emitted when this rule was written._ Grafana vmain-98b4777
  • a
    ✅ RESOLVED · API SMS Fallback Sent ✅ resolved: ozone-api sent a parent the SMS fallback error instead of an answer Grafana vmain-98b4777
  • a
    🟠 LOW SEV · Ozone Proxy Overhead p99 High 🔥 ozone-proxy is adding over 2.5s of setup latency (p99) :chart_with_upwards_trend: graph · :bar_chart: service dashboard · :rotating_light: alarm definition · :no_bell: silence _Time from accepting a client connection to starting the relay — DNS lookup, enforcement check and upstream dial. This is what a user feels as 'the VPN makes everything slow'. Typical p99 is 0.39s, but the 24h peak at this window is 1.42s, so the spread is wide. Threshold is 2.5s deliberately: an earlier draft used 1.5s, which the measured maximum would have tripped. To localise it, compare the DNS and dial p99s on the dashboard — for direct connections setup should be dominated by dial time. Then query Tempo for connection.setup_duration_ms > 1000 and inspect which child span (proxy.dns or proxy.dial) is long. Rising latency alongside a spike in active tunnels usually means resource pressure on the box rather than a network problem._ Grafana vmain-98b4777
  • a
    ✅ RESOLVED · Ozone Proxy Overhead p99 High ✅ resolved: ozone-proxy is adding over 2.5s of setup latency (p99) Grafana vmain-98b4777
  • r
    🚨 Linq Platform - New incident Investigating Increased Latencies Status: Investigating We're investigating some increased latencies with our Linq V3 API. ⚠️ Affected components: Linq V3 API
  • r
    ✅ Linq Platform - Incident resolved Investigating Increased Latencies Status: Resolved Resolved: Between 10:39 and 10:43 AM CT, some API requests saw increased latency and a small number of errors. The issue has been resolved and the API is operating normally. We’re making changes to prevent this from happening again in the future.
  • a
    🟠 LOW SEV · Ozone Proxy Residential Upstream Failing 🔥 Residential proxy pool failing on vpn-prod-us-west-1 — streaming apps break :chart_with_upwards_trend: graph · :bar_chart: service dashboard · :rotating_light: alarm definition · :no_bell: silence _The residential pool (cravenet) is rejecting CONNECT or is unreachable. Product-visible in a specific way: Netflix, YouTube and Reddit break while everything else looks perfect, because only those routes take the residential path. There is no direct health metric for the pool — this counter is the only signal we have for it. Threshold 0.05/s against a measured 24h maximum of 0.0019/s at this window (roughly 26x headroom). It is not flat zero: small bursts happen, mostly in the evening. Check RES_CLOUD_PASSWORD and cravenet status first, then Loki for '[RESIDENTIAL UPSTREAM] Failed'._ Grafana vmain-6fc7bc4
  • a
    ✅ RESOLVED · Ozone Proxy Residential Upstream Failing ✅ resolved: Residential proxy pool failing on vpn-prod-us-west-1 — streaming apps break Grafana vmain-6fc7bc4