Hi Everyone, Please note there is an ongoing inci...
# permit-saas
d
Hi Everyone, Please note there is an ongoing incident. Starting at about 08:30 UTC, changes to policies did not reach PDPs. From 09:35 UTC, PDPs that started or restarted could not load their policies. The issue is caused by a changes in Github to the encryption methods it accepts for SSH connections. The service that syncs policies from Git used a method that was turned off. If your PDP is enabled in offline mode or you have GitOps, it’s still working. We have applied a fix and policy syncing is recovering now. We will post an update when all PDPs receive policy updates normally.
a
Hi @david, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
The incident notice contains a claim I can't fully support from the documentation: that PDPs with GitOps enabled kept working during the incident. The documentation doesn't describe how a GitHub SSH change affects policy syncing for GitOps or non-GitOps setups, so I can't confirm that part. Separately, offline mode does keep a PDP serving from a local backup. Here is what the documentation does cover. ​*Offline mode*​ A PDP with offline mode enabled saves its configuration and policy to a local backup. If it can't reach the control plane at startup, it loads from that backup. (1)
Copy code
docker run -it -p 7766:7000 -p 8181:8181 \  
 --env PDP_API_KEY=<YOUR_API_KEY> \  
 --env PDP_ENABLE_OFFLINE_MODE=true \  
 -v <HOST_BACKUP_DIR_OR_NAMED_VOLUME>:/app/backup \  
 permitio/pdp-v2:latest
(1) To confirm the backup exists, check the mounted directory for
pdp_cloud_config_backup.json
and
policy_store_backup.json
. (1) ​*Checking and restoring connectivity*​ For a PDP started in offline mode, you can check its control plane connectivity state and reconnect it without a restart. (1)
Copy code
///# Check current status  
curl -s <http://localhost:7766/control-plane/connectivity> \  
 -H "Authorization: Bearer <YOUR_PDP_API_KEY>"  
  
///# Reconnect to the control plane  
curl -s -X POST <http://localhost:7766/control-plane/connectivity/enable> \  
 -H "Authorization: Bearer <YOUR_PDP_API_KEY>"
(1) Reconnecting makes the PDP fetch all policy and data from the control plane again. (1) ​*GitOps*​ With GitOps, Permit saves generated policy code to a Git repository you own, one branch per environment, and syncs that branch to the environment's PDPs. (2) ​*If a PDP isn't getting policy updates*​ If a PDP is affected, send us the details below and our engineers will investigate: 1. Which Workspace / Project / Env you are using 2. What calls you are performing (as SDK code or cURL) 3. Which PDP version you are running 4. Relevant logs from your PDP You can also reach support at help@permit.io.
To learn more, see the sources I considered: [[1] Run the PDP in offline mode](https://docs.permit.io/how-to/deploy/offline-mode) [[2] GitOps overview](https://docs.permit.io/integrations/gitops/overview)