Hi team I am trying to follow this guide to spin u...
# permit-saas
j
Hi team I am trying to follow this guide to spin up a PDP in gcp cloud run - https://docs.permit.io/how-to/deploy/cloud-hosts/gcp-cloud-run/. When I do a health check on the instance, it returns 503
Copy code
{
  "components": {
    "horizon": {
      "details": {
        "watchdog": {
          "message": "Watchdog status when direct Horizon health check was not Ok.",
          "status": "error"
        }
      },
      "error": "Failed to connect to Horizon: error sending request for url (<http://0.0.0.0:7001/healthy>)",
      "status": "error"
    },
    "opa": {
      "error": "Failed to connect to OPA: error sending request for url (<http://localhost:8181/health>)",
      "status": "error"
    }
  },
  "status": "error"
}
Is there something wrong/missing in my configs?
a
Hi @Jaieu Sheil, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
a
Your health check shows both Horizon (internal Python service) and OPA are failing to start. Based on the error, here are the key configuration points to check: Container Port Configuration The PDP runs on port 7000 by default (1) . Make sure you've set the container port to 7000 in Cloud Run . Required Environment Variable The only required environment variable is
PDP_API_KEY
(1) (2) . Ensure you've set this with a valid Environment-level API Key (2) . Common Issues to Check 1. ​*CPU Throttling*​: A known issue with Cloud Run can cause Signal 6 termination. Add the
<http://run.googleapis.com/cpu-throttling|run.googleapis.com/cpu-throttling>: false
annotation in your YAML configuration (1) . 2. ​*Cold Starts*​: Set "Minimum number of instances" to 1 in the Auto Scaling section to avoid cold starts affecting authorization queries . 3. ​*API Key Issues*​: Verify your
PDP_API_KEY
is valid and can reach the control plane at
<https://api.permit.io>
(2) . 4. ​*Network/DNS Issues*​: The PDP needs to connect to
<http://api.permit.io|api.permit.io>
to fetch its configuration (3) . If running behind a proxy, you may need to configure proxy settings (4) . Debugging Information Needed To help investigate further, please provide: 1. Which Workspace / Project / Env you are using 2. What calls you're performing (as SDK code or CURL) 3. Which PDP version you're running 4. Relevant logs from your PDP container (especially startup logs) The startup logs will show if the PDP is successfully fetching config from the control plane and any OPA initialization errors.
To learn more, see the sources I considered: [[1] Deploying PDP on GCP Cloud Run](https://docs.permit.io/how-to/deploy/cloud-hosts/gcp-cloud-run) [[2] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[3] Docker image errors out - cannot get config from control plane](https://github.com/permitio/PDP/issues/251) [[4] ERROR PROXY SYSTEM CANT BE USED](https://github.com/permitio/opal/discussions/729)
j
I see an error - Process 'python3' terminated with status: signal: 15 (SIGTERM)" - in the startup logs
o
@Ask Inkeep
a
Thanks for tagging me @Or Weis. I'll look through relevant sources and get back to you shortly.
The SIGTERM error you're seeing indicates the Python process (Horizon service) is being terminated. This is often related to resource constraints in Cloud Run. Based on similar issues, here's what to check: ​*Resource Allocation*​ The PDP needs sufficient resources to run both the Horizon service and OPA (1) . Try increasing: • Memory allocation (at least 512MB-1GB recommended) • CPU allocation (at least 1 vCPU) ​*Container Configuration*​ Ensure your container is configured with the correct ports (2) : • Main PDP API port: 7000 (mapped from container port 7766) • Horizon service port: 7001 (internal) • OPA port: 8181 (internal) ​*Environment Variables*​ Double-check these critical settings :
Copy code
PDP_API_KEY=<YOUR_API_KEY>  # Required
PDP_PORT=7766  # Default container port
PDP_HORIZON_PORT=7001  # Horizon service port
​*Health Check Timeouts*​ The Horizon service needs time to initialize. You can adjust these timeouts : •
PDP_HORIZON_STARTUP_DELAY=5
(seconds before health checks begin) •
PDP_HORIZON_HEALTH_CHECK_TIMEOUT=1
(timeout for each check) •
PDP_HEALTHCHECK_TIMEOUT=3
(overall health check timeout) To help investigate further, please share: 1. Which Workspace / Project / Env you're using 2. What SDK calls you're performing 3. Which PDP version you're running (e.g.,
permitio/pdp-v2:latest
) 4. Complete startup logs showing the initialization sequence The full logs will help identify if it's failing during config fetch, OPA initialization, or resource constraints.
To learn more, see the sources I considered: [[1] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266) [[2] Configurations](https://docs.permit.io/concepts/pdp/configuration)
a
Hello. I'm having the exact same problem. I've already tried all of the troubleshooting steps mentioned in this convo, as well as in other resources online, but with no luck. I've also tried running the service in a VPC with a router and a NAT, and not in a VPC, but I get the same error in both cases. I very much suspect that this is a networking/DNS issue in Google Cloud Run, and not an issue with Permit's PDP, as the same PDP image, API key, and config work running locally in Docker. But if someone here is knowledgable about what could be the issue I'd greatly appreciate it. I've asked Google and ChatGPT, and copied logs to them, but to no avail. I have also tried using different Permit API keys. Here are some other logs that may be helpful (from the Google Cloud Run service's log page): • Service 'python3' health check failed: HTTP request failed: error sending request for url (http://0.0.0.0:7001/healthy) • Caused by: client error (Connect) • Caused by: tcp connect error • Caused by: Connection refused (os error 111) (consecutive failures: 8/12)
➕ 1
o
Hi @Adam Kopp I’ll ask @Eli Moshkovich to invetigate further when he is back from the weekend Can you share you current setup / logs ? : 1. Which Workspace / Project / Env you’re using 2. Which PDP version you’re running (e.g.,
permitio/pdp-v2:latest
) 3. Complete startup logs showing the initialization sequence and error CC: @Ze'ev Manilovich
a
Hi Or, Thanks for the response. I've tried using both my project level API key and the API key for my dev environment. Ideally this would serve as a single PDP for both dev and staging, at least, but just dev is ok, as well. I've tried both permitio/pdp-v2:latest and manually entering permitio/pdp-v2:0.9.9 - both have caused the same issues. Maybe the best way to share my setup with you is to show you my terraform files that I'm using to setup my VPC and the Cloud Run service? What's the best way to upload those? Logs are below. Thanks,
Copy code
DEFAULT 2025-11-30T20:16:39.909735Z 2025-11-30T20:16:39.910473+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:40.304640Z 2025-11-30T20:16:40.303393+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:16:40.304903Z 2025-11-30T20:16:40.303775+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:16:41.696916Z 2025-11-30T20:16:41.697585+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:16:41.697176Z 2025-11-30T20:16:41.697989+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:16:41.979147Z 2025-11-30T20:16:41.977761+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:41.997547Z 2025-11-30T20:16:41.996078+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:42.119491Z 2025-11-30T20:16:42.120340+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:42.141616Z 2025-11-30T20:16:42.142567+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:43.253374Z 2025-11-30T20:16:43.253873+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:43.274922Z 2025-11-30T20:16:43.275424+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:43.286227Z 2025-11-30T20:16:43.284583+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:43.305321Z 2025-11-30T20:16:43.303794+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:43.456902Z 2025-11-30T20:16:43.457136+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:43.479161Z 2025-11-30T20:16:43.479542+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:45.304858Z 2025-11-30T20:16:45.302990+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:16:45.305109Z 2025-11-30T20:16:45.303524+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:16:46.300250Z 2025-11-30T20:16:46.301410+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:46.320242Z 2025-11-30T20:16:46.321495+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:46.696624Z 2025-11-30T20:16:46.697970+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:16:46.697015Z 2025-11-30T20:16:46.698395+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:16:47.124452Z 2025-11-30T20:16:47.125879+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:47.146662Z 2025-11-30T20:16:47.148157+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:48.799934Z 2025-11-30T20:16:48.801032+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:48.821060Z 2025-11-30T20:16:48.822207+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:50.305132Z 2025-11-30T20:16:50.302868+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:16:50.305749Z 2025-11-30T20:16:50.303256+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:16:51.696705Z 2025-11-30T20:16:51.697452+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:16:51.696977Z 2025-11-30T20:16:51.697830+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:16:53.337928Z 2025-11-30T20:16:53.338264+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:53.359350Z 2025-11-30T20:16:53.359768+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:55.137604Z 2025-11-30T20:16:55.135513+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:16:55.154228Z 2025-11-30T20:16:55.152130+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
DEFAULT 2025-11-30T20:16:55.305229Z 2025-11-30T20:16:55.303147+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:16:55.305346Z 2025-11-30T20:16:55.303512+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:16:55.305617Z [2025-11-30T20:16:55Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 6/12)
DEFAULT 2025-11-30T20:16:56.697646Z 2025-11-30T20:16:56.697686+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:16:56.697917Z 2025-11-30T20:16:56.698276+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:17:00.305047Z 2025-11-30T20:17:00.303827+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:17:00.305183Z 2025-11-30T20:17:00.304185+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:35456 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:17:00.305564Z [2025-11-30T20:17:00Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 7/12)
DEFAULT 2025-11-30T20:17:01.696109Z 2025-11-30T20:17:01.697672+0000 | opal_client.policy_store.opa_client |WARNING | OPA client health: False (policy: False, data: False)
DEFAULT 2025-11-30T20:17:01.696462Z 2025-11-30T20:17:01.698135+0000 | uvicorn.protocols.http.httptools_impl | INFO | 127.0.0.1:23987 - "GET /healthy HTTP/1.1" 503
DEFAULT 2025-11-30T20:17:01.697057Z [2025-11-30T20:17:01Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 6/12)
DEFAULT 2025-11-30T20:17:03.402978Z 2025-11-30T20:17:03.401345+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - <wss://opal.permit.io/ws>
DEFAULT 2025-11-30T20:17:03.418599Z 2025-11-30T20:17:03.417174+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - [Errno -5] Name has no usable address
o
Yeah the Terraform file if you can share it would probably help
a
Ok, the terraform files are below. I should also mention that I tried following the directions in https://docs.permit.io/how-to/deploy/cloud-hosts/gcp-cloud-run/ exactly (to stand up the service via the GCP web GUI, not in a VPC or anything) and got the exact same errors. vpc.tf
Copy code
resource "google_compute_network" "main" {
  name = "${local.project_name}-vpc"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "private" {
  name          = "private-subnet"
  network       = google_compute_network.main.id
  ip_cidr_range = "10.0.0.0/28"
  region        = local.region
}

resource "google_vpc_access_connector" "serverless" {
  name   = local.project_name
  region = local.region
  network = google_compute_network.main.name

  ip_cidr_range = "10.0.1.0/28"

  min_throughput = 200
  max_throughput = 300
}

resource "google_compute_router" "main" {
  name    = "main-router"
  region  = local.region
  network = google_compute_network.main.id
}

resource "google_compute_router_nat" "main" {
  name                               = "main-nat"
  router                             = google_compute_router.main.name
  region                             = local.region
  nat_ip_allocate_option             = "AUTO_ONLY"

  # REQUIRED: NAT the connector's range manually
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

  log_config {
    enable = false
    filter = "ERRORS_ONLY"
  }
}
pdp_service.tf
Copy code
resource "google_cloud_run_v2_service" "pdp" {
  name     = "pdp"
  location = local.region
  deletion_protection = false

  template {
    scaling {
      min_instance_count = 1
      max_instance_count = 3
    }

    containers {
      image = var.pdp_image

      resources {
        limits = {
          memory = "1Gi"
          cpu    = "2"
        }
        cpu_idle = false
      }

      ports {
        container_port = 7000
      }

      env {
        name  = "PDP_API_KEY"
        value = local.permit_project_api_key
      }

      env {
        name = "PDP_DEBUG"
        value = "True"
      }

      startup_probe {
        tcp_socket {
          port = 7000
        }
        initial_delay_seconds = 45
        timeout_seconds       = 3
        period_seconds        = 10
        failure_threshold     = 3
      }

      liveness_probe {
        http_get {
          path = "/healthy"
          port = 7000
        }
        initial_delay_seconds = 45
        timeout_seconds       = 3
        period_seconds        = 10
        failure_threshold     = 3
      }
    }
    
    vpc_access {
      connector = google_vpc_access_connector.serverless.id
      egress    = "ALL_TRAFFIC"
    }

  }
  
  ingress = "INGRESS_TRAFFIC_ALL"
}
j
@Adam Kopp @Eli Moshkovich Just curious, were you able to resolve this issue?
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
a
No, still haven't found a solution
e
Hi @Jaieu Sheil @Adam Kopp, Just letting you know that we opened an internal issue for this and we’ll investigate it further. We’ll update here once we have more information.
a
Sounds good, thanks!
j
@Eli Moshkovich Thanks
t
Hey @Jaieu Sheil I just tried and was able to deploy successful via the CLI. These are the steps. You can see the health check here: https://permit-pdp-618750305032.us-central1.run.app/health
Note, this exposes it to the public (which isn't reccomended for production) but it does launch correctly and give a valid health check.
j
Thanks for looking into it. How should we solve this for production scenario?
t
it should be an internal route to the apps
not publicly exposed
a
I tried this and am still not having any luck. @Thomas - is this considered the fix to your internal ticket, or are you guys still investigating?
a
Hey @Adam Kopp , will make sure @Thomas see it and will circle back to you
a
Thanks so much
t
@Adam Kopp https://permit-pdp-618750305032.us-central1.run.app/health my cloud run PDP is returning healthy so seems like a configuration issue in your gcp. Is there anything I can provide from my Cloud Run that will help debug on your end?
a
Hi Thomas. Are you connecting the service to a vpc at all? I've tried both connecting it to a vpc and not. Do you have any firewall rules that you're setting? My best guess is that it's a connection issue that isn't allowing the pdp service to reach out to the API.
t
you mean VPC -> cloud run call?
don't quite know what you mean by connecting to a vpc
a
I mean using a vpc connector to connect the cloud run service to a vpc (cloud run services don't run in vpcs, so if you want a private connection to other resources in the VPC you have to create a connector and run the service's traffic through it). I wasn't having any luck with the VPC connector, so to debug, I tried running the service totally standalone, as in the example. That didn't work either. So, whether using a vpc connector or not, do you have any firewall rules that you set?
j
Hi @Thomas, our org policies in gcp don't allow deployment with vpc-access-egress set to null and ingress set to all so I deployed the service with the allowed settings keeping all other details as mentioned in the above instructions but I still get health check error. Attached is a snapshot from deployment logs
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
a
@Thomas I just copied your cli setup instructions line by line, replacing values for my API key and service account email. I'm still getting all of the same errors and logs as I documented above.
t
@Adam Kopp no firewall rules except allow all ingress, but happy to apply your rules and test from VPC.
hmm that's strange.
I deployed a second with the script 😅
a
where did you set that allow all ingress firewall rule? in the service itself?
t
yes. gcloud run services add-iam-policy-binding permit-pdp \ --region=us-central1 \ --member="allUsers" \ --role="roles/run.invoker"
a
yeah i ran that, too
t
almost has to be an ingress rule on the container somewhere
a
im actually thinking it's an egress rule. i'm able to hit the container just fine and it returns this message:
Copy code
{
  "components": {
    "horizon": {
      "details": {
        "watchdog": {
          "message": "Watchdog status when direct Horizon health check was not Ok.",
          "status": "error"
        }
      },
      "error": "Horizon returned status 503 Service Unavailable",
      "status": "error"
    },
    "opa": {
      "error": null,
      "status": "ok"
    }
  },
  "status": "error"
}
t
can you get into the container and run some tests?
:8181 is the internal opa instance that is timing out
a
i can run it locally and exec into it, but i don't think cloud run lets you do that.
also it works for me locally
here are the logs
Copy code
2025-12-08 15:40:33.240 EST
[32m2025-12-08T20:40:33.241535+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:31956 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:38.239 EST
[32m2025-12-08T20:40:38.240636+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:38.240 EST
[32m2025-12-08T20:40:38.241179+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:31956 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:38.240 EST
[2025-12-08T20:40:38Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 6/12)
2025-12-08 15:40:43.240 EST
[2025-12-08T20:40:43Z WARN watchdog::service] Service 'python3' health check failed: HTTP request failed: error sending request for url (<http://0.0.0.0:7001/healthy>)
2025-12-08 15:40:43.240 EST
2025-12-08 15:40:43.240 EST
Caused by: client error (SendRequest)
2025-12-08 15:40:43.240 EST
2025-12-08 15:40:43.240 EST
Caused by: connection closed before message completed (consecutive failures: 7/12)
2025-12-08 15:40:47.051 EST
GET40187 B1 msChrome 142 <https://permit-pdp-553151235065.us-central1.run.app/>
2025-12-08 15:40:47.054 EST
[2025-12-08T20:40:47Z WARN pdp_server::api::authn_middleware] Missing Authorization header
2025-12-08 15:40:47.138 EST
GET40187 B1 msChrome 142 <https://permit-pdp-553151235065.us-central1.run.app/favicon.ico>
2025-12-08 15:40:47.141 EST
[2025-12-08T20:40:47Z WARN pdp_server::api::authn_middleware] Missing Authorization header
2025-12-08 15:40:48.240 EST
[32m2025-12-08T20:40:48.241169+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:48.240 EST
[32m2025-12-08T20:40:48.241610+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:63960 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:48.240 EST
[2025-12-08T20:40:48Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 8/12)
2025-12-08 15:40:51.527 EST
GET503367 B5 msChrome 142 <https://permit-pdp-553151235065.us-central1.run.app/health>
2025-12-08 15:40:51.531 EST
[32m2025-12-08T20:40:51.532380+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:51.531 EST
[32m2025-12-08T20:40:51.532913+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:62200 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:51.532 EST
[32m2025-12-08T20:40:51.533705+0000[0m | [34mopal_client.engine.logger [0m|[1m INFO | Received request. [38;2;153;153;153mGET /health[0m[1m[0m
2025-12-08 15:40:51.533 EST
[2025-12-08T20:40:51Z INFO pdp_server::api::health::handlers] Health check failed: horizon: Horizon returned status 503 Service Unavailable
2025-12-08 15:40:51.533 EST
[32m2025-12-08T20:40:51.534432+0000[0m | [34mopal_client.engine.logger [0m|[1m INFO | Sent response. [38;2;153;153;153mGET /health -> 200[0m[1m[0m
2025-12-08 15:40:51.627 EST
GET40187 B1 msChrome 142 <https://permit-pdp-553151235065.us-central1.run.app/favicon.ico>
2025-12-08 15:40:51.630 EST
[2025-12-08T20:40:51Z WARN pdp_server::api::authn_middleware] Missing Authorization header
2025-12-08 15:40:53.239 EST
[32m2025-12-08T20:40:53.240671+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:53.239 EST
[32m2025-12-08T20:40:53.241144+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:63960 - "GET /healthy HTTP/1.1" 503[0m
2025-12-08 15:40:53.240 EST
[2025-12-08T20:40:53Z WARN watchdog::service] Service 'python3' health check failed: Unhealthy status code: 503 (expected 200) (consecutive failures: 9/12)
2025-12-08 15:40:54.162 EST
[32m2025-12-08T20:40:54.163517+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
2025-12-08 15:40:54.187 EST
[32m2025-12-08T20:40:54.188769+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
2025-12-08 15:40:57.895 EST
[32m2025-12-08T20:40:57.896908+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | Trying server - <wss://opal.permit.io/ws>[0m
2025-12-08 15:40:57.920 EST
[32m2025-12-08T20:40:57.921295+0000[0m | [34mfastapi_websocket_rpc.websocket_rpc_c...[0m|[1m INFO | RPC Connection failed - [Errno -5] Name has no usable address[0m
2025-12-08 15:40:58.240 EST
[32m2025-12-08T20:40:58.241361+0000[0m | [34mopal_client.policy_store.opa_client [0m|[33m[1mWARNING | OPA client health: False (policy: False, data: False)[0m
2025-12-08 15:40:58.240 EST
[32m2025-12-08T20:40:58.241824+0000[0m | [34muvicorn.protocols.http.httptools_impl [0m|[1m INFO | 127.0.0.1:63960 - "GET /healthy HTTP/1.1" 503[0m
t
@Adam Kopp i wonder if you can set to
--vpc-egress=private-ranges-only
i see the first problem is to OPAL (egress so you were right)
a
ok i figured it out. i was using my project level API key. when i switched to my environment level api key it now works.
are you not able to use a project level key? i thought i read somewhere that you can
t
ah! nice find. i was nowhere near that haha
a
well i tried going back and forth between the two keys early on, but that didn't fix it (must have been another problem at that point)
t
you
can
.... but not the common path
a
ok well thanks. i'll try getting some other infrastructure spun up and see if this works out. appreciate the help
t
my pleasure! glad to see it's working and was something simple
j
I am using the env specific key of the test project, but still get this error
This ^ error is a new one though where is it trying to connect to the pdp control plane and then failing
t
@Adam Kopp is yours still working? something where we can diff the two?
a
yeah mine is still working
what do you need? the yaml?
j
I can compare the two if you don't mind sharing your yaml file @Adam Kopp . I am suspecting in my case there some corp firewall that's causing it
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
a
Here you go, hope this helps:
Copy code
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: pdp-2
  namespace: REDACTED
  selfLink: /apis/serving.knative.dev/v1/namespaces/REDACTED/services/pdp-2
  uid: REDACTED
  resourceVersion: AAZFeSOQpcw
  generation: 1
  creationTimestamp: '2025-12-08T23:26:51.728719Z'
  labels:
    goog-terraform-provisioned: 'true'
    <http://run.googleapis.com/satisfiesPzs|run.googleapis.com/satisfiesPzs>: 'true'
    <http://cloud.googleapis.com/location|cloud.googleapis.com/location>: us-east4
  annotations:
    <http://run.googleapis.com/ingress|run.googleapis.com/ingress>: all
    serving.knative.dev/creator: REDACTED
    serving.knative.dev/lastModifier: REDACTED
    <http://run.googleapis.com/operation-id|run.googleapis.com/operation-id>: REDACTED
    <http://run.googleapis.com/ingress-status|run.googleapis.com/ingress-status>: all
    <http://run.googleapis.com/urls|run.googleapis.com/urls>: '["<https://pdp-2-REDACTED.us-east4.run.app>","<https://pdp-2-REDACTED.a.run.app>"]'
spec:
  template:
    metadata:
      labels:
        <http://run.googleapis.com/startupProbeType|run.googleapis.com/startupProbeType>: Custom
      annotations:
        <http://run.googleapis.com/sessionAffinity|run.googleapis.com/sessionAffinity>: 'false'
        <http://run.googleapis.com/vpc-access-egress|run.googleapis.com/vpc-access-egress>: all-traffic
        autoscaling.knative.dev/minScale: '1'
        autoscaling.knative.dev/maxScale: '3'
        <http://run.googleapis.com/vpc-access-connector|run.googleapis.com/vpc-access-connector>: projects/REDACTED/locations/us-east4/connectors/REDACTED
        <http://run.googleapis.com/cpu-throttling|run.googleapis.com/cpu-throttling>: 'false'
    spec:
      containerConcurrency: 160
      timeoutSeconds: 300
      serviceAccountName: REDACTED
      containers:
      - image: permitio/pdp-v2:latest
        ports:
        - name: http1
          containerPort: 7000
        env:
        - name: PDP_API_KEY
          valueFrom:
            secretKeyRef:
              key: latest
              name: permit-pdp-api-key
        - name: PDP_DEBUG
          value: 'True'
        resources:
          limits:
            cpu: '2'
            memory: 1Gi
        startupProbe:
          initialDelaySeconds: 30
          timeoutSeconds: 3
          periodSeconds: 10
          failureThreshold: 2
          tcpSocket:
            port: 7000
  traffic:
  - percent: 100
    latestRevision: true
status:
  observedGeneration: 1
  conditions:
  - type: Ready
    status: 'True'
    lastTransitionTime: '2025-12-08T23:27:30.305484Z'
  - type: ConfigurationsReady
    status: 'True'
    lastTransitionTime: '2025-12-08T23:26:53.599386Z'
  - type: RoutesReady
    status: 'True'
    lastTransitionTime: '2025-12-08T23:27:30.278058Z'
  latestReadyRevisionName: pdp-2-00001-n7k
  latestCreatedRevisionName: pdp-2-00001-n7k
  traffic:
  - revisionName: pdp-2-00001-n7k
    percent: 100
    latestRevision: true
  url: <https://pdp-2-REDACTED.a.run.app>
  address:
    url: <https://pdp-2-REDACTED.a.run.app>
p
Hey, thanks for reaching out ! It seems our team is a little busy right now, a human will be with you within the next hour or so. In the meantime, can I help you ?
a
and here's the terraform code i used to create it
Copy code
resource "google_cloud_run_v2_service" "pdp" {
  name     = "pdp-2"
  location = local.region
  deletion_protection = false

  template {
    scaling {
      min_instance_count = 1
      max_instance_count = 3
    }

    service_account = google_service_account.cloud_run_deploy.email

    containers {
      image = var.pdp_image

      resources {
        limits = {
          memory = "1Gi"
          cpu    = "2"
        }
        cpu_idle = false
      }

      ports {
        name = "http1"
        container_port = 7000
      }

      env {
        name  = "PDP_API_KEY"
        value_source {
          secret_key_ref {
            secret = google_secret_manager_secret.permit_pdp_api_key.secret_id
            version = "latest"
          }
        }
      }

      env {
        name = "PDP_DEBUG"
        value = "True"
      }

      startup_probe {
        tcp_socket {
          port = 7000
        }
        initial_delay_seconds = 30
        timeout_seconds       = 3
        period_seconds        = 10
        failure_threshold     = 2
      }
    }
    
    vpc_access {
      connector = google_vpc_access_connector.serverless.id
      egress    = "ALL_TRAFFIC"
    }

  }
  depends_on = [ 
    google_compute_network.main,
    google_compute_subnetwork.private,
    google_service_networking_connection.private_vpc_connection
  ]
  ingress = "INGRESS_TRAFFIC_ALL"
}

resource "google_compute_network" "main" {
  name = "${local.project_name}-vpc"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "private" {
  name          = "private-subnet"
  network       = google_compute_network.main.id
  ip_cidr_range = "10.0.0.0/28"
  region        = local.region
}

resource "google_compute_address" "internal_with_subnet_and_address" {
  name         = "my-internal-address"
  subnetwork   = google_compute_subnetwork.private.id
  address_type = "INTERNAL"
  address      = "10.0.0.10"
  region       = local.region
}

resource "google_vpc_access_connector" "serverless" {
  name   = local.project_name
  region = local.region
  project = local.google_project_id

  subnet {
    name = google_compute_subnetwork.private.name
  }

  # Required: max_throughput or max_instances - ßchoose one
  min_throughput = 200
  max_throughput = 300  # Mbps (cheapest option)
}

resource "google_compute_router" "main" {
  name    = "main-router"
  region  = local.region
  network = google_compute_network.main.id
}

resource "google_compute_router_nat" "main" {
  name                               = "main-nat"
  router                             = google_compute_router.main.name
  region                             = local.region
  nat_ip_allocate_option             = "AUTO_ONLY"

  # REQUIRED: NAT the connector's range manually
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

  log_config {
    enable = false
    filter = "ERRORS_ONLY"
  }
}

# 1. Reserve an IP range for Private Service Access
resource "google_compute_global_address" "private_service_range" {
  name          = "private-service-range"
  purpose       = "VPC_PEERING"
  address_type  = "INTERNAL"
  prefix_length = 16
  network       = google_compute_network.main.id
}

# 2. Create the Private Service Connection
resource "google_service_networking_connection" "private_vpc_connection" {
  network = google_compute_network.main.id
  service = "<http://servicenetworking.googleapis.com|servicenetworking.googleapis.com>"

  reserved_peering_ranges = [
    google_compute_global_address.private_service_range.name
  ]
}

resource "google_service_account" "cloud_run_deploy" {
  account_id = "cloud-run-deploy"
  display_name = "Cloud Run Deploy"
  project = local.google_project_id
}

resource "google_project_iam_member" "service_account_storage_admin" {
  role               = "roles/storage.admin"
  member             = "serviceAccount:${google_service_account.cloud_run_deploy.email}"
  project = local.google_project_id
}

resource "google_project_iam_member" "service_account_user" {
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:${google_service_account.cloud_run_deploy.email}"
  project = local.google_project_id
}

resource "google_project_iam_member" "service_account_run_admin" {
  role               = "roles/run.admin"
  member             = "serviceAccount:${google_service_account.cloud_run_deploy.email}"
  project = local.google_project_id
}

resource "google_project_iam_member" "service_account_secret_accessor" {
  role               = "roles/secretmanager.secretAccessor"
  member             = "serviceAccount:${google_service_account.cloud_run_deploy.email}"
  project = local.google_project_id
}