Another question here. idk if i'm doing this wrong or not. but few questions. • when we host our own local pdp (k8s sidecar) is it supposed to be eventual consistency with the cloud? • We've seen some inconsistent things where permit cloud will have accurate roles/permissions for a user, but when we go to check via our app -> pdp sidecar they're denied. The time of their roles/permissions being applied was at least 1.5 hrs ago, yet they were still denied. I needed to run

/data-updater/trigger

to force our sidecars to resync with cloud. Is there like an obvious configuration in yml or something I may be overlooking for this?

Hi @Engineering Swantide, I'm an AI assistant for Permit.io. I'll analyze available sources and get back to you shortly.

Hi @Engineering Swantide the Agent gave a nice answer. If after confirming what it suggests, it still does not sync properly can you please share 1) your workspace/project/env 2) the curl requests you are making 3) relevant pdp logs I can take a look into it

Thanks — sharing the full bundle: 1. Workspace / Project / Env

- Workspace (org): b7566fbe41224674a6e5d2b754bc092a
  - Project: 9b41011e6c514c30bbb8bc8bdb069386
  - Environment (Production): ed27c8ee868e4b34b7174bca5c22b5e7
  - PDP instance id (from OPAL logs): 9779e761-9fb1-481c-aab6-a6b45237c323

2. PDP version + deployment

- permitio/pdp-v2:0.9.10, running as a native sidecar (Kubernetes initContainer with restartPolicy: Always)
  alongside our backend service in private GKE (us-west2). Egress through GCP Cloud NAT with default TCP
  timeouts (tcpEstablishedIdleTimeoutSec=1200s).
  - PDP env: PDP_API_KEY (set), PDP_DEBUG=false, OPAL_LOG_SERIALIZE=True, OPAL_LOG_FORMAT="". No OPAL keepalive
  / data refresh / WS-ping env vars set — all defaults.
  - PDP container has been running uninterrupted since 2026-05-09T19:15:15Z (restartCount 0).

3. The call we make from our backend

// Server-side check via PDP sidecar at <http://localhost:7000>
  await fetch('<http://localhost:7000/allowed>', {
    method: 'POST',
    headers: { 'Content-Type':'application/json', Authorization: `Bearer ${PERMIT_API_KEY}` },
    body: JSON.stringify({
      user:   { key: '6a0380dff8e2a9c2899dcf26' },
      action: 'read',
      resource: { type: 'organization', key: '69e26f54c4a1bc9fd385c6f2', tenant: 'default' }
    })
  })

Equivalent curl:

curl -X POST <http://localhost:7000/allowed> \
    -H "Authorization: Bearer $PERMIT_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"user":{"key":"6a0380dff8e2a9c2899dcf26"},"action":"read","resource":{"type":"organization","key":"69e2
  6f54c4a1bc9fd385c6f2","tenant":"default"}}'

4. Timeline (UTC, 2026-05-12) - 193457 — permit.api.roleAssignments.bulkAssign succeeds in cloud Permit, creating 3 instance-role assignments for user 6a0380dff8e2a9c2899dcf26: - User on organization:69e26f54c4a1bc9fd385c6f2 - User on metadata_documentation:69e26f54c4a1bc9fd385c6f2 - User on ai_assistant:69e26f54c4a1bc9fd385c6f2 - All in tenant default. Confirmed via GET /v2/facts/.../role_assignments?user=... (returned all 3). - 193455 — sidecar OPAL receives notification Insert on User by id='6aa3f796-986e-4124-b732-4ae09b197ffb', fetches from pdp-deltas.api.permit.io/.../users/6aa3f796986e4124b7324ae09b197ffb, writes PUT /v1/data/users/6a0380dff8e2a9c2899dcf26 → 204. - 194207 — second Update on User event for same user, same outcome. - No notification with dst_path for relationships/organization, resource_instances/organization, or role_assignments/* for this user ever arrived between 193457 and ~20:53 (we have full OPAL logs for this window if helpful). - 193501, 194102, 194431 — sidecar /allowed returns { "allow": false, "allowing_sources": [] } for the check above. - Cloud PDP (https://cloudpdp.api.permit.io/allowed) returned allow: true with rebac.allow: true, reason: "granted by 'organization:69e26f54c4a1bc9fd385c6f2#User'" the entire time, confirming cloud had the assignment correctly. - ~20:53 — we ran POST /data-updater/trigger on each of our 4 sidecar PDP pods. All sidecars then return allow: true for the same input. 5. What we've verified from our side - We don't write to OPA directly. All Permit writes go through the official Node SDK (permit.api.roleAssignments.bulkAssign, etc). - Same drift was present on all 4 sidecar replicas — not a one-pod issue. - We saw a previous user (6a038323f8e2a9c2899dcf29, same org, assigned at 194437Z) with the same symptom; data-updater/trigger resolved that one too. - No reconnect / disconnect / connection-error log lines in the PDP logs around the incident at our current verbosity (PDP_DEBUG=false). What we'd love your help with - From your side, was a policy_data delta event for the role_assignment (or relationship/resource_instance) for user 6a0380dff8e2a9c2899dcf26 actually published to PDP instance 9779e761-9fb1-481c-aab6-a6b45237c323 between 193457 and 20:53 UTC? - If yes — your end shows it sent but our PDP didn't apply it. We can enable PDP_DEBUG=true on a rolled pod for next time and capture WS-level logs. - If no — that pins it on the cloud-side delta publisher and we'd appreciate confirmation that a fix is in flight. - Either way: is there a recommended OPAL_* env var combo for keepalive that we should set under GCP Cloud NAT (1200s TCP idle) to keep the long-lived WS warm? Right now we run pure defaults.

Thanks for the detailed writeup. I will look into it and get back to you

thank you!

Thanks for the detailed writeup. I suspect this is a WS half-open issue rather than a cloud-side delta problem. You're running pure OPAL defaults, so OPAL_KEEP_ALIVE_INTERVAL=0 (keepalives off). GCP Cloud NAT drops idle TCP flows at 1200s — if the WS sits idle past that window, the connection looks alive to OPAL but is torn down at the NAT, so subsequent deltas never land and no reconnect fires.

/data-updater/trigger

resolves it because it's a separate HTTP path. Matches every symptom you described, including no reconnect logs. Try setting

OPAL_KEEP_ALIVE_INTERVAL=30

Also it couldnt hurt to update to latest PDP with

PDP_DEBUG=true

so we get WS-level logs if it recurs.

ok thats where we landed on too, thanks for verifying that