Jaieu Sheil
11/26/2025, 10:00 PM{
"components": {
"horizon": {
"details": {
"watchdog": {
"message": "Watchdog status when direct Horizon health check was not Ok.",
"status": "error"
}
},
"error": "Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>)",
"status": "error"
},
"opa": {
"error": "Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)",
"status": "error"
}
},
"status": "error"
}
Is there something wrong/missing in my configs?Ask Inkeep
11/26/2025, 10:00 PMpermit support automation2
11/26/2025, 10:00 PMAsk Inkeep
11/26/2025, 10:01 PMPDP_API_KEY (1) (2) . Ensure you've set this with a valid Environment-level API Key (2) .
Common Issues to Check
1. *CPU Throttling*: A known issue with Cloud Run can cause Signal 6 termination. Add the <http://run.googleapis.com/cpu-throttling|run.googleapis.com/cpu-throttling>: false annotation in your YAML configuration (1) .
2. *Cold Starts*: Set "Minimum number of instances" to 1 in the Auto Scaling section to avoid cold starts affecting authorization queries .
3. *API Key Issues*: Verify your PDP_API_KEY is valid and can reach the control plane at <https://api.permit.io> (2) .
4. *Network/DNS Issues*: The PDP needs to connect to <http://api.permit.io|api.permit.io> to fetch its configuration (3) . If running behind a proxy, you may need to configure proxy settings (4) .
Debugging Information Needed
To help investigate further, please provide:
1. Which Workspace / Project / Env you are using
2. What calls you're performing (as SDK code or CURL)
3. Which PDP version you're running
4. Relevant logs from your PDP container (especially startup logs)
The startup logs will show if the PDP is successfully fetching config from the control plane and any OPA initialization errors.Ask Inkeep
11/26/2025, 10:01 PMJaieu Sheil
11/26/2025, 10:03 PMOr Weis
11/27/2025, 6:21 AMAsk Inkeep
11/27/2025, 6:21 AMAsk Inkeep
11/27/2025, 6:21 AMPDP_API_KEY=<YOUR_API_KEY> # Required
PDP_PORT=7766 # Default container port
PDP_HORIZON_PORT=7001 # Horizon service port
*Health Check Timeouts*
The Horizon service needs time to initialize. You can adjust these timeouts :
• PDP_HORIZON_STARTUP_DELAY=5 (seconds before health checks begin)
• PDP_HORIZON_HEALTH_CHECK_TIMEOUT=1 (timeout for each check)
• PDP_HEALTHCHECK_TIMEOUT=3 (overall health check timeout)
To help investigate further, please share:
1. Which Workspace / Project / Env you're using
2. What SDK calls you're performing
3. Which PDP version you're running (e.g., permitio/pdp-v2:latest)
4. Complete startup logs showing the initialization sequence
The full logs will help identify if it's failing during config fetch, OPA initialization, or resource constraints.Ask Inkeep
11/27/2025, 6:22 AMAdam Kopp
11/30/2025, 6:54 PMOr Weis
11/30/2025, 7:52 PMpermitio/pdp-v2:latest)
3. Complete startup logs showing the initialization sequence and error
CC: @Ze'ev ManilovichAdam Kopp
11/30/2025, 8:09 PMDEFAULT 2025-11-30T20:16:39.909735Z [32m2025-11-30T20:16:39.910473+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:40.304640Z [32m2025-11-30T20:16:40.303393+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:16:40.304903Z [32m2025-11-30T20:16:40.303775+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:16:41.696916Z [32m2025-11-30T20:16:41.697585+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:16:41.697176Z [32m2025-11-30T20:16:41.697989+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:16:41.979147Z [32m2025-11-30T20:16:41.977761+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:41.997547Z [32m2025-11-30T20:16:41.996078+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:42.119491Z [32m2025-11-30T20:16:42.120340+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:42.141616Z [32m2025-11-30T20:16:42.142567+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:43.253374Z [32m2025-11-30T20:16:43.253873+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:43.274922Z [32m2025-11-30T20:16:43.275424+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:43.286227Z [32m2025-11-30T20:16:43.284583+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:43.305321Z [32m2025-11-30T20:16:43.303794+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:43.456902Z [32m2025-11-30T20:16:43.457136+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:43.479161Z [32m2025-11-30T20:16:43.479542+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:45.304858Z [32m2025-11-30T20:16:45.302990+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:16:45.305109Z [32m2025-11-30T20:16:45.303524+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:16:46.300250Z [32m2025-11-30T20:16:46.301410+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:46.320242Z [32m2025-11-30T20:16:46.321495+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:46.696624Z [32m2025-11-30T20:16:46.697970+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:16:46.697015Z [32m2025-11-30T20:16:46.698395+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:16:47.124452Z [32m2025-11-30T20:16:47.125879+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:47.146662Z [32m2025-11-30T20:16:47.148157+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:48.799934Z [32m2025-11-30T20:16:48.801032+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:48.821060Z [32m2025-11-30T20:16:48.822207+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:50.305132Z [32m2025-11-30T20:16:50.302868+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:16:50.305749Z [32m2025-11-30T20:16:50.303256+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:16:51.696705Z [32m2025-11-30T20:16:51.697452+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:16:51.696977Z [32m2025-11-30T20:16:51.697830+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:16:53.337928Z [32m2025-11-30T20:16:53.338264+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:53.359350Z [32m2025-11-30T20:16:53.359768+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:55.137604Z [32m2025-11-30T20:16:55.135513+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:16:55.154228Z [32m2025-11-30T20:16:55.152130+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
DEFAULT 2025-11-30T20:16:55.305229Z [32m2025-11-30T20:16:55.303147+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:16:55.305346Z [32m2025-11-30T20:16:55.303512+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:16:55.305617Z [2025-11-30T20:16:55Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 6/12)
DEFAULT 2025-11-30T20:16:56.697646Z [32m2025-11-30T20:16:56.697686+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:16:56.697917Z [32m2025-11-30T20:16:56.698276+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:17:00.305047Z [32m2025-11-30T20:17:00.303827+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:17:00.305183Z [32m2025-11-30T20:17:00.304185+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:17:00.305564Z [2025-11-30T20:17:00Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 7/12)
DEFAULT 2025-11-30T20:17:01.696109Z [32m2025-11-30T20:17:01.697672+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
DEFAULT 2025-11-30T20:17:01.696462Z [32m2025-11-30T20:17:01.698135+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503[0m
DEFAULT 2025-11-30T20:17:01.697057Z [2025-11-30T20:17:01Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 6/12)
DEFAULT 2025-11-30T20:17:03.402978Z [32m2025-11-30T20:17:03.401345+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
DEFAULT 2025-11-30T20:17:03.418599Z [32m2025-11-30T20:17:03.417174+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0mOr Weis
11/30/2025, 8:30 PMAdam Kopp
11/30/2025, 8:37 PMresource "google_compute_network" "main" {
name = "${local.project_name}-vpc"
auto_create_subnetworks = false
}
resource "google_compute_subnetwork" "private" {
name = "private-subnet"
network = google_compute_network.main.id
ip_cidr_range = "10.0.0.0/28"
region = local.region
}
resource "google_vpc_access_connector" "serverless" {
name = local.project_name
region = local.region
network = google_compute_network.main.name
ip_cidr_range = "10.0.1.0/28"
min_throughput = 200
max_throughput = 300
}
resource "google_compute_router" "main" {
name = "main-router"
region = local.region
network = google_compute_network.main.id
}
resource "google_compute_router_nat" "main" {
name = "main-nat"
router = google_compute_router.main.name
region = local.region
nat_ip_allocate_option = "AUTO_ONLY"
# REQUIRED: NAT the connector's range manually
source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
log_config {
enable = false
filter = "ERRORS_ONLY"
}
}
pdp_service.tf
resource "google_cloud_run_v2_service" "pdp" {
name = "pdp"
location = local.region
deletion_protection = false
template {
scaling {
min_instance_count = 1
max_instance_count = 3
}
containers {
image = var.pdp_image
resources {
limits = {
memory = "1Gi"
cpu = "2"
}
cpu_idle = false
}
ports {
container_port = 7000
}
env {
name = "PDP_API_KEY"
value = local.permit_project_api_key
}
env {
name = "PDP_DEBUG"
value = "True"
}
startup_probe {
tcp_socket {
port = 7000
}
initial_delay_seconds = 45
timeout_seconds = 3
period_seconds = 10
failure_threshold = 3
}
liveness_probe {
http_get {
path = "/healthy"
port = 7000
}
initial_delay_seconds = 45
timeout_seconds = 3
period_seconds = 10
failure_threshold = 3
}
}
vpc_access {
connector = google_vpc_access_connector.serverless.id
egress = "ALL_TRAFFIC"
}
}
ingress = "INGRESS_TRAFFIC_ALL"
}Jaieu Sheil
12/03/2025, 9:03 PMpermit support automation2
12/03/2025, 9:03 PMAdam Kopp
12/03/2025, 9:18 PMEli Moshkovich
12/03/2025, 9:29 PMAdam Kopp
12/03/2025, 9:31 PMJaieu Sheil
12/04/2025, 5:47 PMThomas
12/04/2025, 7:20 PMThomas
12/04/2025, 7:22 PMJaieu Sheil
12/04/2025, 7:30 PMThomas
12/04/2025, 8:19 PMThomas
12/04/2025, 8:19 PMAdam Kopp
12/07/2025, 3:46 PMAlon Boshi
12/07/2025, 6:05 PMAdam Kopp
12/07/2025, 6:21 PMThomas
12/08/2025, 2:58 PMAdam Kopp
12/08/2025, 3:16 PMThomas
12/08/2025, 3:18 PMThomas
12/08/2025, 3:18 PMAdam Kopp
12/08/2025, 3:47 PMJaieu Sheil
12/08/2025, 8:20 PMpermit support automation2
12/08/2025, 8:20 PMAdam Kopp
12/08/2025, 8:29 PMThomas
12/08/2025, 8:31 PMThomas
12/08/2025, 8:31 PMThomas
12/08/2025, 8:32 PMAdam Kopp
12/08/2025, 8:32 PMThomas
12/08/2025, 8:33 PMAdam Kopp
12/08/2025, 8:36 PMThomas
12/08/2025, 8:37 PMAdam Kopp
12/08/2025, 8:38 PM{
"components": {
"horizon": {
"details": {
"watchdog": {
"message": "Watchdog status when direct Horizon health check was not Ok.",
"status": "error"
}
},
"error": "Horizon returned status 503 Service Unavailable",
"status": "error"
},
"opa": {
"error": null,
"status": "ok"
}
},
"status": "error"
}Thomas
12/08/2025, 8:38 PMThomas
12/08/2025, 8:39 PMAdam Kopp
12/08/2025, 8:39 PMAdam Kopp
12/08/2025, 8:39 PMAdam Kopp
12/08/2025, 8:42 PM2025-12-08 15:40:33.240 EST
[32m2025-12-08T20:40:33.241535+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:31956 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:38.239 EST
[32m2025-12-08T20:40:38.240636+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:38.240 EST
[32m2025-12-08T20:40:38.241179+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:31956 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:38.240 EST
[2025-12-08T20:40:38Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 6/12)
2025-12-08 15:40:43.240 EST
[2025-12-08T20:40:43Z WARN watchdog::service] Service 'python3' health check failed: HTTP request failed: error sending request for url (<http://0.0.0.0:7001/healthy>)
2025-12-08 15:40:43.240 EST
2025-12-08 15:40:43.240 EST
Caused by: client error (SendRequest)
2025-12-08 15:40:43.240 EST
2025-12-08 15:40:43.240 EST
Caused by: connection closed before message completed (consecutive failures: 7/12)
2025-12-08 15:40:47.051 EST
GET40187 B1 msChrome 142 <https://permit-pdp-553151235065.us-central1.run.app/>
2025-12-08 15:40:47.054 EST
[2025-12-08T20:40:47Z WARN pdp_server::api::authn_middleware] Missing Authorization header
2025-12-08 15:40:47.138 EST
GET40187 B1 msChrome 142 <https://permit-pdp-553151235065.us-central1.run.app/favicon.ico>
2025-12-08 15:40:47.141 EST
[2025-12-08T20:40:47Z WARN pdp_server::api::authn_middleware] Missing Authorization header
2025-12-08 15:40:48.240 EST
[32m2025-12-08T20:40:48.241169+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:48.240 EST
[32m2025-12-08T20:40:48.241610+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:63960 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:48.240 EST
[2025-12-08T20:40:48Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 8/12)
2025-12-08 15:40:51.527 EST
GET503367 B5 msChrome 142 <https://permit-pdp-553151235065.us-central1.run.app/health>
2025-12-08 15:40:51.531 EST
[32m2025-12-08T20:40:51.532380+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:51.531 EST
[32m2025-12-08T20:40:51.532913+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:62200 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:51.532 EST
[32m2025-12-08T20:40:51.533705+0000[0m | [34mopal_client.engine.logger [0m|[1m INFO | Received request. [38;2;153;153;153mGET /health[0m[1m[0m
2025-12-08 15:40:51.533 EST
[2025-12-08T20:40:51Z INFO pdp_server::api::health::handlers] Health check failed: horizon: Horizon returned status 503 Service Unavailable
2025-12-08 15:40:51.533 EST
[32m2025-12-08T20:40:51.534432+0000[0m | [34mopal_client.engine.logger [0m|[1m INFO | Sent response. [38;2;153;153;153mGET /health -> 200[0m[1m[0m
2025-12-08 15:40:51.627 EST
GET40187 B1 msChrome 142 <https://permit-pdp-553151235065.us-central1.run.app/favicon.ico>
2025-12-08 15:40:51.630 EST
[2025-12-08T20:40:51Z WARN pdp_server::api::authn_middleware] Missing Authorization header
2025-12-08 15:40:53.239 EST
[32m2025-12-08T20:40:53.240671+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:53.239 EST
[32m2025-12-08T20:40:53.241144+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:63960 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:53.240 EST
[2025-12-08T20:40:53Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 9/12)
2025-12-08 15:40:54.162 EST
[32m2025-12-08T20:40:54.163517+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
2025-12-08 15:40:54.187 EST
[32m2025-12-08T20:40:54.188769+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
2025-12-08 15:40:57.895 EST
[32m2025-12-08T20:40:57.896908+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
2025-12-08 15:40:57.920 EST
[32m2025-12-08T20:40:57.921295+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
2025-12-08 15:40:58.240 EST
[32m2025-12-08T20:40:58.241361+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:58.240 EST
[32m2025-12-08T20:40:58.241824+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:63960 - "GET /healthy HTTP/1.1" 503[0mThomas
12/08/2025, 8:51 PM--vpc-egress=private-ranges-onlyThomas
12/08/2025, 8:52 PMAdam Kopp
12/08/2025, 8:52 PMAdam Kopp
12/08/2025, 8:53 PMThomas
12/08/2025, 8:53 PMAdam Kopp
12/08/2025, 8:53 PMThomas
12/08/2025, 8:53 PMcan .... but not the common pathAdam Kopp
12/08/2025, 8:54 PMThomas
12/08/2025, 8:55 PMJaieu Sheil
12/08/2025, 8:55 PMJaieu Sheil
12/08/2025, 8:56 PMThomas
12/08/2025, 9:06 PMAdam Kopp
12/08/2025, 9:07 PMAdam Kopp
12/08/2025, 9:09 PMJaieu Sheil
12/08/2025, 10:28 PMpermit support automation2
12/08/2025, 10:28 PMAdam Kopp
12/08/2025, 11:33 PMapiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: pdp-2
namespace: REDACTED
selfLink: /apis/serving.knative.dev/v1/namespaces/REDACTED/services/pdp-2
uid: REDACTED
resourceVersion: AAZFeSOQpcw
generation: 1
creationTimestamp: '2025-12-08T23:26:51.728719Z'
labels:
goog-terraform-provisioned: 'true'
<http://run.googleapis.com/satisfiesPzs|run.googleapis.com/satisfiesPzs>: 'true'
<http://cloud.googleapis.com/location|cloud.googleapis.com/location>: us-east4
annotations:
<http://run.googleapis.com/ingress|run.googleapis.com/ingress>: all
serving.knative.dev/creator: REDACTED
serving.knative.dev/lastModifier: REDACTED
<http://run.googleapis.com/operation-id|run.googleapis.com/operation-id>: REDACTED
<http://run.googleapis.com/ingress-status|run.googleapis.com/ingress-status>: all
<http://run.googleapis.com/urls|run.googleapis.com/urls>: '["<https://pdp-2-REDACTED.us-east4.run.app>","<https://pdp-2-REDACTED.a.run.app>"]'
spec:
template:
metadata:
labels:
<http://run.googleapis.com/startupProbeType|run.googleapis.com/startupProbeType>: Custom
annotations:
<http://run.googleapis.com/sessionAffinity|run.googleapis.com/sessionAffinity>: 'false'
<http://run.googleapis.com/vpc-access-egress|run.googleapis.com/vpc-access-egress>: all-traffic
autoscaling.knative.dev/minScale: '1'
autoscaling.knative.dev/maxScale: '3'
<http://run.googleapis.com/vpc-access-connector|run.googleapis.com/vpc-access-connector>: projects/REDACTED/locations/us-east4/connectors/REDACTED
<http://run.googleapis.com/cpu-throttling|run.googleapis.com/cpu-throttling>: 'false'
spec:
containerConcurrency: 160
timeoutSeconds: 300
serviceAccountName: REDACTED
containers:
- image: permitio/pdp-v2:latest
ports:
- name: http1
containerPort: 7000
env:
- name: PDP_API_KEY
valueFrom:
secretKeyRef:
key: latest
name: permit-pdp-api-key
- name: PDP_DEBUG
value: 'True'
resources:
limits:
cpu: '2'
memory: 1Gi
startupProbe:
initialDelaySeconds: 30
timeoutSeconds: 3
periodSeconds: 10
failureThreshold: 2
tcpSocket:
port: 7000
traffic:
- percent: 100
latestRevision: true
status:
observedGeneration: 1
conditions:
- type: Ready
status: 'True'
lastTransitionTime: '2025-12-08T23:27:30.305484Z'
- type: ConfigurationsReady
status: 'True'
lastTransitionTime: '2025-12-08T23:26:53.599386Z'
- type: RoutesReady
status: 'True'
lastTransitionTime: '2025-12-08T23:27:30.278058Z'
latestReadyRevisionName: pdp-2-00001-n7k
latestCreatedRevisionName: pdp-2-00001-n7k
traffic:
- revisionName: pdp-2-00001-n7k
percent: 100
latestRevision: true
url: <https://pdp-2-REDACTED.a.run.app>
address:
url: <https://pdp-2-REDACTED.a.run.app>permit support automation2
12/08/2025, 11:33 PMAdam Kopp
12/08/2025, 11:37 PMresource "google_cloud_run_v2_service" "pdp" {
name = "pdp-2"
location = local.region
deletion_protection = false
template {
scaling {
min_instance_count = 1
max_instance_count = 3
}
service_account = google_service_account.cloud_run_deploy.email
containers {
image = var.pdp_image
resources {
limits = {
memory = "1Gi"
cpu = "2"
}
cpu_idle = false
}
ports {
name = "http1"
container_port = 7000
}
env {
name = "PDP_API_KEY"
value_source {
secret_key_ref {
secret = google_secret_manager_secret.permit_pdp_api_key.secret_id
version = "latest"
}
}
}
env {
name = "PDP_DEBUG"
value = "True"
}
startup_probe {
tcp_socket {
port = 7000
}
initial_delay_seconds = 30
timeout_seconds = 3
period_seconds = 10
failure_threshold = 2
}
}
vpc_access {
connector = google_vpc_access_connector.serverless.id
egress = "ALL_TRAFFIC"
}
}
depends_on = [
google_compute_network.main,
google_compute_subnetwork.private,
google_service_networking_connection.private_vpc_connection
]
ingress = "INGRESS_TRAFFIC_ALL"
}
resource "google_compute_network" "main" {
name = "${local.project_name}-vpc"
auto_create_subnetworks = false
}
resource "google_compute_subnetwork" "private" {
name = "private-subnet"
network = google_compute_network.main.id
ip_cidr_range = "10.0.0.0/28"
region = local.region
}
resource "google_compute_address" "internal_with_subnet_and_address" {
name = "my-internal-address"
subnetwork = google_compute_subnetwork.private.id
address_type = "INTERNAL"
address = "10.0.0.10"
region = local.region
}
resource "google_vpc_access_connector" "serverless" {
name = local.project_name
region = local.region
project = local.google_project_id
subnet {
name = google_compute_subnetwork.private.name
}
# Required: max_throughput or max_instances - ßchoose one
min_throughput = 200
max_throughput = 300 # Mbps (cheapest option)
}
resource "google_compute_router" "main" {
name = "main-router"
region = local.region
network = google_compute_network.main.id
}
resource "google_compute_router_nat" "main" {
name = "main-nat"
router = google_compute_router.main.name
region = local.region
nat_ip_allocate_option = "AUTO_ONLY"
# REQUIRED: NAT the connector's range manually
source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"
log_config {
enable = false
filter = "ERRORS_ONLY"
}
}
# 1. Reserve an IP range for Private Service Access
resource "google_compute_global_address" "private_service_range" {
name = "private-service-range"
purpose = "VPC_PEERING"
address_type = "INTERNAL"
prefix_length = 16
network = google_compute_network.main.id
}
# 2. Create the Private Service Connection
resource "google_service_networking_connection" "private_vpc_connection" {
network = google_compute_network.main.id
service = "<http://servicenetworking.googleapis.com|servicenetworking.googleapis.com>"
reserved_peering_ranges = [
google_compute_global_address.private_service_range.name
]
}
resource "google_service_account" "cloud_run_deploy" {
account_id = "cloud-run-deploy"
display_name = "Cloud Run Deploy"
project = local.google_project_id
}
resource "google_project_iam_member" "service_account_storage_admin" {
role = "roles/storage.admin"
member = "serviceAccount:${google_service_account.cloud_run_deploy.email}"
project = local.google_project_id
}
resource "google_project_iam_member" "service_account_user" {
role = "roles/iam.serviceAccountUser"
member = "serviceAccount:${google_service_account.cloud_run_deploy.email}"
project = local.google_project_id
}
resource "google_project_iam_member" "service_account_run_admin" {
role = "roles/run.admin"
member = "serviceAccount:${google_service_account.cloud_run_deploy.email}"
project = local.google_project_id
}
resource "google_project_iam_member" "service_account_secret_accessor" {
role = "roles/secretmanager.secretAccessor"
member = "serviceAccount:${google_service_account.cloud_run_deploy.email}"
project = local.google_project_id
}