[FIRING:1] WireGuard No Active Peers critical (production Production Infrastructure) Firing Value: C=1 Labels: - alertname = WireGuard No Active Peers - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/cfhtll6bmq5fkf/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dcfhtll6bmq5fkf&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[FIRING:1] WireGuard Active Peer Count Drop critical (production Production Infrastructure) Firing Value: C=1 Labels: - alertname = WireGuard Active Peer Count Drop - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = No WireGuard peers have completed a handshake in the past 5 minutes on production. The fleet appears fully offline. Possible causes: WireGuard interface down, nftables change, server reboot, network outage, or wireguard-exporter dead. - summary = WireGuard active peer count critically low Source: https://clearspace.grafana.net/alerting/grafana/bfpj5pbzzjeo0b/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dbfpj5pbzzjeo0b&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[FIRING:1] DatasourceNoData critical (grafanacloud-prom production Production Infrastructure A Ozone Proxy Down) Firing Value: [no value] Labels: - alertname = DatasourceNoData - datasource_uid = grafanacloud-prom - environment = production - grafana_folder = Production Infrastructure - ref_id = A - rulename = Ozone Proxy Down - severity = critical Annotations: - description = ozone-proxy metrics endpoint is unreachable. All client TCP traffic (including DNS via mock server) is failing — total outage for connected clients. - summary = Ozone Proxy is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/ffpj5pbzi248wd/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffpj5pbzi248wd&matcher=datasource_uid%3Dgrafanacloud-prom&matcher=environment%3Dproduction&matcher=ref_id%3DA&matcher=rulename%3DOzone+Proxy+Down&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[FIRING:1] WireGuard No Active Peers critical (production Production Infrastructure) Firing Value: C=1 Labels: - alertname = WireGuard No Active Peers - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/cfhtll6bmq5fkf/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dcfhtll6bmq5fkf&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[FIRING:1] WireGuard Active Peer Count Drop critical (production Production Infrastructure) Firing Value: C=1 Labels: - alertname = WireGuard Active Peer Count Drop - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = No WireGuard peers have completed a handshake in the past 5 minutes on production. The fleet appears fully offline. Possible causes: WireGuard interface down, nftables change, server reboot, network outage, or wireguard-exporter dead. - summary = WireGuard active peer count critically low Source: https://clearspace.grafana.net/alerting/grafana/bfpj5pbzzjeo0b/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dbfpj5pbzzjeo0b&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[FIRING:1] DatasourceNoData critical (grafanacloud-prom production Production Infrastructure A Ozone Proxy Down) Firing Value: [no value] Labels: - alertname = DatasourceNoData - datasource_uid = grafanacloud-prom - environment = production - grafana_folder = Production Infrastructure - ref_id = A - rulename = Ozone Proxy Down - severity = critical Annotations: - description = ozone-proxy metrics endpoint is unreachable. All client TCP traffic (including DNS via mock server) is failing — total outage for connected clients. - summary = Ozone Proxy is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/ffpj5pbzi248wd/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffpj5pbzi248wd&matcher=datasource_uid%3Dgrafanacloud-prom&matcher=environment%3Dproduction&matcher=ref_id%3DA&matcher=rulename%3DOzone+Proxy+Down&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[FIRING:2] DatasourceNoData warning (grafanacloud-prom production Production Infrastructure A) Firing Value: [no value] Labels: - alertname = DatasourceNoData - datasource_uid = grafanacloud-prom - environment = production - grafana_folder = Production Infrastructure - ref_id = A - rulename = WireGuard Exporter Down - severity = warning Annotations: - description = prometheus-wireguard-exporter is unreachable. WireGuard peer metrics are not being collected — handshake loop and peer offline alerts will not fire. - summary = WireGuard Exporter is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/afpj5pbzx1ibkf/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dafpj5pbzx1ibkf&matcher=datasource_uid%3Dgrafanacloud-prom&matcher=environment%3Dproduction&matcher=ref_id%3DA&matcher=rulename%3DWireGuard+Exporter+Down&matcher=severity%3Dwarning&orgId=1 Value: [no value] Labels: - alertname = DatasourceNoData - datasource_uid = grafanacloud-prom - environment = production - grafana_folder = Production Infrastructure - ref_id = A - rulename = WireGuard Config Manager Down - severity = warning Annotations: - description = wireguard-config-manager is unreachable. New devices cannot connect and peer cleanup is not running. - summary = WireGuard Config Manager is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/efpj5pbzujlz4f/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Defpj5pbzujlz4f&matcher=datasource_uid%3Dgrafanacloud-prom&matcher=environment%3Dproduction&matcher=ref_id%3DA&matcher=rulename%3DWireGuard+Config+Manager+Down&matcher=severity%3Dwarning&orgId=1 Grafana vmain-b6060fe

[RESOLVED] WireGuard Active Peer Count Drop critical (production Production Infrastructure) Resolved Value: C=0 Labels: - alertname = WireGuard Active Peer Count Drop - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = No WireGuard peers have completed a handshake in the past 5 minutes on production. The fleet appears fully offline. Possible causes: WireGuard interface down, nftables change, server reboot, network outage, or wireguard-exporter dead. - summary = WireGuard active peer count critically low Source: https://clearspace.grafana.net/alerting/grafana/bfpj5pbzzjeo0b/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dbfpj5pbzzjeo0b&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[RESOLVED] WireGuard No Active Peers critical (production Production Infrastructure) Resolved Value: C=0 Labels: - alertname = WireGuard No Active Peers - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/cfhtll6bmq5fkf/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dcfhtll6bmq5fkf&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[RESOLVED] DatasourceNoData critical (grafanacloud-prom production Production Infrastructure A Ozone Proxy Down) Resolved Value: [no value] Labels: - alertname = DatasourceNoData - datasource_uid = grafanacloud-prom - environment = production - grafana_folder = Production Infrastructure - ref_id = A - rulename = Ozone Proxy Down - severity = critical Annotations: - description = ozone-proxy metrics endpoint is unreachable. All client TCP traffic (including DNS via mock server) is failing — total outage for connected clients. - grafana_state_reason = MissingSeries - summary = Ozone Proxy is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/ffpj5pbzi248wd/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffpj5pbzi248wd&matcher=datasource_uid%3Dgrafanacloud-prom&matcher=environment%3Dproduction&matcher=ref_id%3DA&matcher=rulename%3DOzone+Proxy+Down&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[RESOLVED] DatasourceNoData warning (grafanacloud-prom production Production Infrastructure A) Resolved Value: [no value] Labels: - alertname = DatasourceNoData - datasource_uid = grafanacloud-prom - environment = production - grafana_folder = Production Infrastructure - ref_id = A - rulename = WireGuard Exporter Down - severity = warning Annotations: - description = prometheus-wireguard-exporter is unreachable. WireGuard peer metrics are not being collected — handshake loop and peer offline alerts will not fire. - grafana_state_reason = MissingSeries - summary = WireGuard Exporter is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/afpj5pbzx1ibkf/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dafpj5pbzx1ibkf&matcher=datasource_uid%3Dgrafanacloud-prom&matcher=environment%3Dproduction&matcher=ref_id%3DA&matcher=rulename%3DWireGuard+Exporter+Down&matcher=severity%3Dwarning&orgId=1 Value: [no value] Labels: - alertname = DatasourceNoData - datasource_uid = grafanacloud-prom - environment = production - grafana_folder = Production Infrastructure - ref_id = A - rulename = WireGuard Config Manager Down - severity = warning Annotations: - description = wireguard-config-manager is unreachable. New devices cannot connect and peer cleanup is not running. - grafana_state_reason = MissingSeries - summary = WireGuard Config Manager is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/efpj5pbzujlz4f/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Defpj5pbzujlz4f&matcher=datasource_uid%3Dgrafanacloud-prom&matcher=environment%3Dproduction&matcher=ref_id%3DA&matcher=rulename%3DWireGuard+Config+Manager+Down&matcher=severity%3Dwarning&orgId=1 Grafana vmain-b6060fe

[FIRING:1] Archive Inserter No Successful Run warning (info staging staging Staging Infrastructure ozone-vpn-staging-us-west-1 systemd-journal systemd-journal archive-inserter.service) Firing Value: B=-1, C=-1 Labels: - alertname = Archive Inserter No Successful Run - detected_level = info - env = staging - environment = staging - grafana_folder = Staging Infrastructure - instance = ozone-vpn-staging-us-west-1 - job = systemd-journal - service_name = systemd-journal - severity = warning - unit = archive-inserter.service Annotations: - Error = [sse.dataQueryError] failed to execute query [A]: no index gateway instances found for tenant 1452025 - description = No 'Run complete' log from archive-inserter.service in the last 15 minutes. The timer may be firing but the inserter is crashing before completion. - grafana_state_reason = Error - summary = Archive inserter has not completed a successful run in 15 minutes Source: https://clearspace.grafana.net/alerting/grafana/ffim4i9hipybka/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffim4i9hipybka&matcher=detected_level%3Dinfo&matcher=env%3Dstaging&matcher=environment%3Dstaging&matcher=instance%3Dozone-vpn-staging-us-west-1&matcher=job%3Dsystemd-journal&matcher=service_name%3Dsystemd-journal&matcher=severity%3Dwarning&matcher=unit%3Darchive-inserter.service&orgId=1 Grafana vmain-b6060fe

[RESOLVED] Archive Inserter No Successful Run warning (info staging staging Staging Infrastructure ozone-vpn-staging-us-west-1 systemd-journal systemd-journal archive-inserter.service) Resolved Value: B=3, C=0 Labels: - alertname = Archive Inserter No Successful Run - detected_level = info - env = staging - environment = staging - grafana_folder = Staging Infrastructure - instance = ozone-vpn-staging-us-west-1 - job = systemd-journal - service_name = systemd-journal - severity = warning - unit = archive-inserter.service Annotations: - description = No 'Run complete' log from archive-inserter.service in the last 15 minutes. The timer may be firing but the inserter is crashing before completion. - summary = Archive inserter has not completed a successful run in 15 minutes Source: https://clearspace.grafana.net/alerting/grafana/ffim4i9hipybka/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffim4i9hipybka&matcher=detected_level%3Dinfo&matcher=env%3Dstaging&matcher=environment%3Dstaging&matcher=instance%3Dozone-vpn-staging-us-west-1&matcher=job%3Dsystemd-journal&matcher=service_name%3Dsystemd-journal&matcher=severity%3Dwarning&matcher=unit%3Darchive-inserter.service&orgId=1 Grafana vmain-b6060fe

[FIRING:1] WireGuard No Active Peers critical (staging Staging Infrastructure) Firing Value: C=1 Labels: - alertname = WireGuard No Active Peers - environment = staging - grafana_folder = Staging Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/ffhtll01afdhca/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffhtll01afdhca&matcher=environment%3Dstaging&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[RESOLVED] WireGuard No Active Peers critical (staging Staging Infrastructure) Resolved Value: C=0 Labels: - alertname = WireGuard No Active Peers - environment = staging - grafana_folder = Staging Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/ffhtll01afdhca/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffhtll01afdhca&matcher=environment%3Dstaging&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[FIRING:1] WireGuard No Active Peers critical (staging Staging Infrastructure) Firing Value: C=1 Labels: - alertname = WireGuard No Active Peers - environment = staging - grafana_folder = Staging Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/ffhtll01afdhca/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffhtll01afdhca&matcher=environment%3Dstaging&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[RESOLVED] WireGuard No Active Peers critical (staging Staging Infrastructure) Resolved Value: C=0 Labels: - alertname = WireGuard No Active Peers - environment = staging - grafana_folder = Staging Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/ffhtll01afdhca/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffhtll01afdhca&matcher=environment%3Dstaging&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[FIRING:1] Ozone Proxy Down critical (production Production Infrastructure) Firing Value: C=1 Labels: - alertname = Ozone Proxy Down - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = ozone-proxy metrics endpoint is unreachable. All client TCP traffic (including DNS via mock server) is failing — total outage for connected clients. - summary = Ozone Proxy is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/ffpj5pbzi248wd/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffpj5pbzi248wd&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

[RESOLVED] Ozone Proxy Down critical (production Production Infrastructure) Resolved Value: C=0 Labels: - alertname = Ozone Proxy Down - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = ozone-proxy metrics endpoint is unreachable. All client TCP traffic (including DNS via mock server) is failing — total outage for connected clients. - summary = Ozone Proxy is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/ffpj5pbzi248wd/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffpj5pbzi248wd&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-b6060fe

🚨 VPN silent-wedge alert: Silent-wedge classification: tunnel broken vs no network

🚨 VPN silent-wedge alert: Silent-wedge classification: tunnel broken vs no network

🚨 VPN silent-wedge alert: Silent-wedge classification: tunnel broken vs no network

🚨 VPN silent-wedge alert: Silent-wedge classification: tunnel broken vs no network

[FIRING:1] Ozone Proxy Down critical (production Production Infrastructure) Firing Value: C=1 Labels: - alertname = Ozone Proxy Down - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = ozone-proxy metrics endpoint is unreachable. All client TCP traffic (including DNS via mock server) is failing — total outage for connected clients. - summary = Ozone Proxy is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/ffpj5pbzi248wd/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffpj5pbzi248wd&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-a25d658

[RESOLVED] Ozone Proxy Down critical (production Production Infrastructure) Resolved Value: C=0 Labels: - alertname = Ozone Proxy Down - environment = production - grafana_folder = Production Infrastructure - severity = critical Annotations: - description = ozone-proxy metrics endpoint is unreachable. All client TCP traffic (including DNS via mock server) is failing — total outage for connected clients. - summary = Ozone Proxy is down on [no value] Source: https://clearspace.grafana.net/alerting/grafana/ffpj5pbzi248wd/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffpj5pbzi248wd&matcher=environment%3Dproduction&matcher=severity%3Dcritical&orgId=1 Grafana vmain-a25d658

[FIRING:1] WireGuard No Active Peers critical (staging Staging Infrastructure) Firing Value: C=1 Labels: - alertname = WireGuard No Active Peers - environment = staging - grafana_folder = Staging Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/ffhtll01afdhca/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffhtll01afdhca&matcher=environment%3Dstaging&matcher=severity%3Dcritical&orgId=1 Grafana vmain-a25d658

[RESOLVED] WireGuard No Active Peers critical (staging Staging Infrastructure) Resolved Value: C=0 Labels: - alertname = WireGuard No Active Peers - environment = staging - grafana_folder = Staging Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/ffhtll01afdhca/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffhtll01afdhca&matcher=environment%3Dstaging&matcher=severity%3Dcritical&orgId=1 Grafana vmain-a25d658

[FIRING:1] WireGuard No Active Peers critical (staging Staging Infrastructure) Firing Value: C=1 Labels: - alertname = WireGuard No Active Peers - environment = staging - grafana_folder = Staging Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/ffhtll01afdhca/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffhtll01afdhca&matcher=environment%3Dstaging&matcher=severity%3Dcritical&orgId=1 Grafana vmain-a25d658

[RESOLVED] WireGuard No Active Peers critical (staging Staging Infrastructure) Resolved Value: C=0 Labels: - alertname = WireGuard No Active Peers - environment = staging - grafana_folder = Staging Infrastructure - severity = critical Annotations: - description = No peers have completed a handshake in the last 5 minutes on [no value] - summary = No active WireGuard peers Source: https://clearspace.grafana.net/alerting/grafana/ffhtll01afdhca/view?orgId=1 Silence: https://clearspace.grafana.net/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dffhtll01afdhca&matcher=environment%3Dstaging&matcher=severity%3Dcritical&orgId=1 Grafana vmain-a25d658

🚨 VPN silent-wedge alert: Silent-wedge classification: tunnel broken vs no network