When running a self-hosted PDP sidecar, there's no built-in rate limiting on the PDP itself (1) . However, timeouts during bulk checks can occur due to query timeout settings. The PDP has a configurable timeout for OPA queries. By default,

PDP_OPA_CLIENT_QUERY_TIMEOUT

is set to

1

second (2) . Bulk checking is more performance-intensive than regular checking and takes more time, which can lead to OPA timeouts (3) . To address timeout issues with bulk checks, you can increase the query timeout by setting this environment variable :

PDP_OPA_CLIENT_QUERY_TIMEOUT=10

(2) , (3) This sets the timeout to 10 seconds. You may also need to add more CPU resources to your container if you're processing large bulk check requests . For your sidecar deployment, there was a known issue where the PDP might try to push policy data before OPA is ready, which was resolved by increasing resources on the sidecar container (4) . To help debug this further, could you provide: 1. Which Workspace / Project / Env you are using 2. What calls you are performing (as SDK code or CURL) 3. Which PDP version you are running (you mentioned v0.9.9) 4. Relevant logs from your PDP during the timeout periods This will help our engineers investigate the intermittent timeout behavior you're experiencing.

To learn more, see the sources I considered: [[1] Cloud PDP Capabilities](https://docs.permit.io/concepts/pdp/cloud-pdp-capabilities) [[2] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[3] Bulk Check](https://docs.permit.io/how-to/enforce-permissions/bulk-check) [[4] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266)

Based on your setup, the intermittent timeouts are likely caused by resource constraints on your PDP container, especially given that you're firing ~18 concurrent

permit.check()

calls during frontend initialization. ​*Key Issues:*​ 1. ​*Default OPA timeout is too low*​: With

PDP_OPA_CLIENT_QUERY_TIMEOUT

defaulting to

1

second (1) , and 18+ concurrent checks hitting OPA simultaneously, you're likely overwhelming the policy engine 2. ​*Concurrent checks vs. bulk checks*​: Using

Promise.all()

over individual

permit.check()

calls creates N separate OPA queries instead of a single optimized bulk operation 3. ​*Resource constraints*​: Your PDP container may not have sufficient CPU/memory allocated for handling concurrent load spikes ​*Recommended Solutions:*​ ​*1. Increase OPA query timeout:*​

PDP_OPA_CLIENT_QUERY_TIMEOUT=5

This gives OPA more time to process queries under load. In PDP v0.9.0 and above,

0

means 0 seconds (not unlimited) . ​*2. Use native bulk check API:*​ Instead of

Promise.all()

over individual checks, use the

/allowed/bulk

endpoint (2) :

const permitted = await permit.bulkCheck([
  { user: userId, action: "read", resource: "contract" },
  { user: userId, action: "create", resource: "document" },
  // ... all 18+ checks
]) ;

This sends a single request to the PDP, which processes all checks more efficiently . ​*3. Monitor PDP resources:*​ Check your PDP container's CPU and memory usage during timeout periods. The PDP includes health check endpoints and metrics capabilities (1) . ​*4. Consider PDP configuration:*​ • `PDP_OPA_CLIENT_FAILURE_THRESHOLD`: Default

0.1

(10% failed requests triggers health check failure) • `PDP_OPA_CLIENT_FAILURE_THRESHOLD_INTERVAL`: Default

60

seconds ​*To help debug further, please share:*​ 1. Which Workspace / Project / Env you are using 2. PDP container resource limits (CPU/memory) 3. PDP logs during timeout periods (especially OPA-related errors) 4. Monitoring data showing CPU/memory usage during failures The combination of increasing the OPA timeout and switching to native bulk checks should significantly improve your timeout issues.

To learn more, see the sources I considered: [[1] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[2] Bulk Check](https://docs.permit.io/how-to/enforce-permissions/bulk-check)

Additionally, if I use the latest version of Permit, i.e., v0.9.13, with the same YAML config as listed above, the docker container never shows up as healthy, however, the permission checks work fine. Why is that?

Regarding your health check issue with PDP v0.9.13, there are a few configuration problems in your YAML: ​*Issues with your current configuration:*​ 1. ​*Invalid timeout configurations*​: The environment variables

PDP_READ_TIMEOUT

,

PDP_WRITE_TIMEOUT

, and

PDP_REQUEST_TIMEOUT

don't exist in PDP configuration (1) . These are not recognized PDP settings. 2. ​*Missing critical timeout*​: You haven't set

PDP_OPA_CLIENT_QUERY_TIMEOUT

, which defaults to

1

second . This is causing your concurrent permission checks to timeout. 3. ​*Health check timing issue*​: There's a known issue where PDPs can be marked as "ready" before all initial data sources are fully loaded (2) (3) . The

/health

endpoint may report healthy before OPA is fully initialized with policy data (4) . ​*Recommended configuration changes:*​

permit-pdp:
    image: permitio/pdp-v2:0.9.13
    container_name: nodeos-permit-pdp-${ENVIRONMENT:-prod}
    restart: unless-stopped
    ports:
      - '${PERMIT_PDP_PORT:-7000}:7000'
    environment:
      - PDP_API_KEY=${PERMIT_API_KEY}
      - PDP_DEBUG=${PDP_DEBUG:-false}
      - PDP_CONTROL_PLANE=${PERMIT_CONTROL_PLANE:-<https://api.permit.io>}
      - PDP_OPA_CLIENT_QUERY_TIMEOUT=5
      - PDP_HEALTHCHECK_TIMEOUT=3
      - PDP_CONTROL_PLANE_TIMEOUT=75
    healthcheck:
      test: ['CMD', 'wget', '--no-verbose', '--tries=1', '--spider', '<http://localhost:7000/health>']
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 60s
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '2.0'
        reservations:
          memory: 1G
          cpus: '1.0'

​*Key changes:*​ • ​*`PDP_OPA_CLIENT_QUERY_TIMEOUT=5`*​: Increases OPA query timeout to 5 seconds (from default 1 second) (1) • ​*`PDP_HEALTHCHECK_TIMEOUT=3`*​: Controls health check timeout (default is 3 seconds) • ​*`PDP_CONTROL_PLANE_TIMEOUT=75`*​: Timeout for control plane requests (default 75 seconds) • ​*Removed invalid timeout variables*​:

PDP_READ_TIMEOUT

,

PDP_WRITE_TIMEOUT

,

PDP_REQUEST_TIMEOUT

• ​*Increased health check intervals*​:

start_period: 60s

and

retries: 10

give the PDP more time to initialize before being marked unhealthy ​*Why health checks might fail on v0.9.13:*​ The PDP's health check endpoint may report unhealthy if the initial data load hasn't completed, even though permission checks work (2) (3) . The Watchdog service monitors internal component health, and stricter health checks were introduced in newer versions (5) . ​*To help debug further, please provide:*​ 1. Which Workspace / Project / Env you are using 2. Complete PDP logs from startup through the health check failures 3. Output from

docker logs nodeos-permit-pdp-${ENVIRONMENT}

showing the initialization sequence 4. Your PDP version (confirm it's exactly v0.9.13) This information will help our engineers investigate why the health check is failing despite permiss

To learn more, see the sources I considered: [[1] Configurations](https://docs.permit.io/concepts/pdp/configuration) [[2] How to detect all OPAL_DATA_CONFIG_SOURCES have been fully loaded at startup](https://github.com/permitio/opal/discussions/887) [[3] /ready healthcheck to be dependant on initial data sources load](https://github.com/permitio/opal/issues/801) [[4] Unable to push the policy data since OPA is not ready yet in sidecar pattern](https://github.com/permitio/PDP/issues/266) [[5] Watchdog](https://github.com/permitio/PDP/blob/main/watchdog/README.md)

Your logs show the PDP is starting correctly—OPA initializes, connects to the control plane, and begins loading policy and data. The health check failure likely stems from timing issues between when Docker checks

/health

and when OPAL finishes its initial data sync. (1) ​*Root cause analysis from your logs:*​ 1. At

07:10:40

OPA reports healthy (

Policy engine is healthy and ready

) 2. At

07:10:41

OPAL starts subscribing to policy and data topics 3. Your health check runs every 20s with a 30s

start_period

, but there's no indication in the logs that the initial data load completed before the first health check ​*Known issue with health checks and data loading:*​ There's a documented issue where the

/health

endpoint can report status before all initial data sources are fully loaded. (2) (3) The

ready

flag in OPAL's health check may remain

false

if initial data loading hasn't completed, even though permission checks work. (2) ​*Recommended workaround:*​ Based on a working solution from the community, use offline mode with backup files to detect when initial sync completes: (3)

permit-pdp:
    image: permitio/pdp-v2:0.9.13
    container_name: nodeos-permit-pdp-${ENVIRONMENT:-prod}
    restart: unless-stopped
    ports:
      - '${PERMIT_PDP_PORT:-7000}:7000'
    environment:
      - PDP_API_KEY=${PERMIT_API_KEY}
      - PDP_DEBUG=${PDP_DEBUG:-false}
      - PDP_CONTROL_PLANE=${PERMIT_CONTROL_PLANE:-<https://api.permit.io>}
      - PDP_OPA_CLIENT_QUERY_TIMEOUT=5
      - PDP_HEALTHCHECK_TIMEOUT=3
      - OPAL_OFFLINE_MODE_ENABLED=true
      - OPAL_STORE_BACKUP_INTERVAL=10
    volumes:
      - pdp-backup:/app/backup
    healthcheck:
      test: ["CMD", "sh", "-c", "wget -q -O /dev/null <http://localhost:7000/health> && test -e /app/backup"]
      interval: 30s
      timeout: 10s
      retries: 10
      start_period: 60s
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '2.0'
        reservations:
          memory: 1G
          cpus: '1.0'

volumes:
  pdp-backup:

​*Key changes:*​ •

OPAL_OFFLINE_MODE_ENABLED=true

enables backup file creation (4) •

OPAL_STORE_BACKUP_INTERVAL=10

creates backup every 10 seconds (default 60s) • Health check includes

test -e /app/backup

to verify initial sync completed (3) • Increased

start_period: 60s

and

retries: 10

give more time for initialization The backup file is written only after the initial sync cycle completes, making its existence a reliable indicator that all data sources have loaded. On subsequent restarts, the persisted backup loads as initial state, eliminating the race condition. ​*To help debug further, please provide:*​ 1. Which Workspace / Project / Env you are using 2. Complete logs showing what happens after the point where your logs cut off (especially any data loading completion messages) 3. Output from

docker exec nodeos-permit-pdp-prod wget -O - <http://localhost:7000/health>

to see the actual health response 4. Confirmation you're running exactly v0.9.13 This workaround should resolve the health check issue while maintaining functional permission checks.

To learn more, see the sources I considered: [[1] Permit's PDP (Policy Decision Point)](https://docs.permit.io/concepts/pdp/overview) [[2] How to detect all OPAL_DATA_CONFIG_SOURCES have been fully loaded at startup](https://github.com/permitio/opal/discussions/887) [[3] /ready healthcheck to be dependant on initial data sources load](https://github.com/permitio/opal/issues/801) [[4] Configurations](https://docs.permit.io/concepts/pdp/configuration)

Great to hear the solutions are working for you! ​*Using Tenants for Multi-Workspace Authorization*​ Yes, using ​*Tenant ID*​ is exactly the right approach for your use case. In Permit.io, tenants represent isolated instances of your application environment, and each tenant can have its own set of users with different roles. (1) Your "workspaces" should map directly to tenants in Permit.io. This allows the same user to have different roles in different workspaces (tenants) . (1) (2) ​*How Multi-Tenant RBAC Works*​ When a user belongs to multiple tenants, their role assignments are scoped per tenant. (2) This means: • A single user can exist in multiple tenants • Role assignments are per tenant • Permissions are evaluated dynamically based on the tenant context ​*Node.js Code Examples*​ ​*1. Syncing a User to Permit*​ First, sync your user to Permit.io: (3)

await permit.api.syncUser({
  key: "key",
  email: "<mailto:example@permit.io|example@permit.io>",
  first_name: "John",
  last_name: "Smith",
  attributes: {},
}) ;

​*2. Assigning Roles to Users in Different Tenants*​ To assign a user different roles in different workspaces (tenants) , use the role assignment API:

await permit.api.roleAssignments.assign({
  user,
  role,
  tenant: "default",
  resource_instance: "document:1",
}) ;

For your specific use case with multiple workspaces, you would assign roles like this:

// Assign user as "admin" in workspace A
await permit.api.roleAssignments.assign({
  user: "user_id",
  role: "admin",
  tenant: "workspace_a",
}) ;

// Assign same user as "viewer" in workspace B
await permit.api.roleAssignments.assign({
  user: "user_id",
  role: "viewer",
  tenant: "workspace_b",
}) ;

​*3. Checking Permissions in a Specific Tenant Context*​ When checking permissions, always specify the tenant to ensure the correct role is evaluated: (3) (4)

const permitted = await permit.check("<mailto:john@permit.io|john@permit.io>", "read", "document") ;

(3) For multi-tenant checks, include the tenant in the resource object:

const permitted = await permit.check(
  "user_id",
  "read",
  {
    type: "document",
    tenant: "workspace_a"
  }
) ;

​*Best Practices*​ • Define your workspaces as tenants in Permit.io (2) • Sync users once, then assign them roles in each tenant they belong to (5) • Always include the tenant context when checking permissions (2) • Users can have completely different roles across tenants without conflict This approach ensures proper isolation between workspaces while allowing users to participate in multiple contexts with appropriate permissions for each.

To learn more, see the sources I considered: [[1] Concepts / Glossary](https://docs.permit.io/overview/glossary) [[2] Best Practices for Multi-Tenant Authorization](https://www.permit.io/blog/best-practices-for-multi-tenant-authorization) [[3] Node.js SDK Quickstart](https://docs.permit.io/sdk/nodejs/quickstart-nodejs) [[4] Node.js All Tenants Check Examples](https://docs.permit.io/sdk/nodejs/all-tenants) [[5] Implement Multi-Tenancy Role-Based Access Control (RBAC) in MongoDB](https://www.permit.io/blog/implement-multi-tenancy-rbac-in-mongodb)

​*Using bulkCheck for Multi-Tenant Permission Checks*​ Yes, you can use

bulkCheck()

to efficiently check multiple permissions for a single user across different tenants. (1) The

bulkCheck()

function accepts an array of check requests, where each request can specify a different tenant context. This is perfect for your multi-workspace scenario where you need to verify what a user can do across their various workspaces. ​*Basic bulkCheck Example*​ Here's how to use

bulkCheck()

for a single user across multiple resources:

const permitted = await permit.bulkCheck([
  { user: "<mailto:anna@smith.com|anna@smith.com>", action: "read", resource: "contract" },
  { user: "<mailto:anna@smith.com|anna@smith.com>", action: "create", resource: "document" },
  { user: "<mailto:john@walker.com|john@walker.com>", action: "sign", resource: "contract" },
  { user: "<mailto:jane@jarvis.com|jane@jarvis.com>", action: "authorize", resource: "invoice" },
]) ;

​*Multi-Tenant bulkCheck Example*​ For your use case with different workspaces (tenants) , you can check permissions across multiple tenants in a single call by including the tenant in the resource object: (1) (2)

const userId = "user_id";

const permissions = await permit.bulkCheck([
  // Check read permission in workspace A
  { 
    user: userId, 
    action: "read", 
    resource: { type: "document", tenant: "workspace_a" } 
  },
  // Check create permission in workspace A
  { 
    user: userId, 
    action: "create", 
    resource: { type: "document", tenant: "workspace_a" } 
  },
  // Check read permission in workspace B
  { 
    user: userId, 
    action: "read", 
    resource: { type: "document", tenant: "workspace_b" } 
  },
  // Check delete permission in workspace B
  { 
    user: userId, 
    action: "delete", 
    resource: { type: "document", tenant: "workspace_b" } 
  }
]) ;

// permissions is an array of booleans corresponding to each check
// permissions[0] = can read documents in workspace_a
// permissions[1] = can create documents in workspace_a
// permissions[2] = can read documents in workspace_b
// permissions[3] = can delete documents in workspace_b

​*Performance Considerations*​ Since bulk checking is more performance-intensive than regular checking, you may need to increase the OPA query timeout if you're checking many permissions at once: (1)

PDP_OPA_CLIENT_QUERY_TIMEOUT=10

This sets a 10-second timeout. You may also need to add more CPU to your container for large bulk check requests. ​*Best Practice for Multi-Workspace Checks*​ When checking permissions for a user across multiple workspaces, always specify the tenant context in each check to ensure the correct role is evaluated for that specific workspace. (2)

To learn more, see the sources I considered: [[1] Bulk Check](https://docs.permit.io/how-to/enforce-permissions/bulk-check) [[2] Implement Multi-Tenancy Role-Based Access Control (RBAC) in MongoDB](https://www.permit.io/blog/implement-multi-tenancy-rbac-in-mongodb)

Hey @Ask Inkeep. So the new changes have been working great so far, at least in local environment. I have noticed something however, while updating a tenant (creation or deletion), I got these logs in permit-pdp container running on Docker.

2026-06-17T22:15:30.915373+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://pdp-deltas.api.permit.io/v2/internal/opal_data/9f83afe1-55e7-49ee-950e-cac413a75f58/6b3e6334-7228-4b2c-bfff-850caf37027e/1b9ddcad-339e-4e4e-84dc-2ec85495d945/users/daefdb31745f40e5b0ae363f8faefc02>

2026-06-17T22:15:30.916221+0000 | opal_client.callbacks.reporter          | INFO  | Reporting the update to requested callbacks

2026-06-17T22:15:30.916522+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://pdp-deltas.api.permit.io/v2/pdps/me/opal_data_callback>

2026-06-17T22:15:30.916964+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://opal-relay.api.permit.io/v2/pdp/callback?retry=0>

2026-06-17T22:15:32.793040+0000 | opal_client.engine.logger               |ERROR  | log upload failed, server replied with HTTP 503 Service Unavailable. {"plugin": "decision_logs", "time": "2026-06-17T22:15:32Z"}

2026-06-17T22:15:33.268325+0000 | uvicorn.protocols.http.httptools_impl   | INFO  | 127.0.0.1:56422 - "GET /healthy HTTP/1.1" 200

2026-06-17T22:15:33.362697+0000 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='<https://pdp-deltas.api.permit.io/v2/internal/opal_data/9f83afe1-55e7-49ee-950e-cac413a75f58/6b3e6334-7228-4b2c-bfff-850caf37027e/1b9ddcad-339e-4e4e-84dc-2ec85495d945/users/daefdb31745f40e5b0ae363f8faefc02>', destination path='/users/68e64179-0082-4665-93a3-0cf9489a5780'

2026-06-17T22:15:33.366235+0000 | opal_client.engine.logger               | INFO  | Received request.    PUT /v1/data/users/68e64179-0082-4665-93a3-0cf9489a5780

2026-06-17T22:15:33.367330+0000 | opal_client.engine.logger               | INFO  | Sent response.       PUT /v1/data/users/68e64179-0082-4665-93a3-0cf9489a5780 -> 204

2026-06-17T22:15:33.369386+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://pdp-deltas.api.permit.io/v2/internal/opal_data/9f83afe1-55e7-49ee-950e-cac413a75f58/6b3e6334-7228-4b2c-bfff-850caf37027e/1b9ddcad-339e-4e4e-84dc-2ec85495d945/users/daefdb31745f40e5b0ae363f8faefc02>

2026-06-17T22:15:33.370006+0000 | opal_client.callbacks.reporter          | INFO  | Reporting the update to requested callbacks

2026-06-17T22:15:33.370245+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://pdp-deltas.api.permit.io/v2/pdps/me/opal_data_callback>

2026-06-17T22:15:33.370510+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://opal-relay.api.permit.io/v2/pdp/callback?retry=0>

2026-06-17T22:15:33.712686+0000 | opal_client.data.fetcher                |ERROR  | Timeout while fetching url: <https://pdp-deltas.api.permit.io/v2/internal/opal_data/9f83afe1-55e7-49ee-950e-cac413a75f58/6b3e6334-7228-4b2c-bfff-850caf37027e/1b9ddcad-339e-4e4e-84dc-2ec85495d945/role_assignments/daefdb31745f40e5b0ae363f8faefc02>

Traceback (most recent call last):

  File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait

    await fut

asyncio.exceptions.CancelledError


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for

    return fut.result()

asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/usr/local/lib/python3.10/site-packages/opal_client/data/fetcher.py", line 75, in handle_url

    response = await self._engine.handle_url(url, config=config)

  File "/usr/local/lib/python3.10/site-packages/opal_common/fetcher/engine/fetching_engine.py", line 116, in handle_url

    await asyncio.wait_for(wait_event.wait(), timeout)

  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for

    raise exceptions.TimeoutError() from exc

asyncio.exceptions.TimeoutError

2026-06-17T22:15:33.713752+0000 | opal_client.data.updater                |ERROR  | Failed to fetch data for entry url='<https://pdp-deltas.api.permit.io/v2/internal/opal_data/9f83afe1-55e7-49ee-950e-cac413a75f58/6b3e6334-7228-4b2c-bfff-850caf37027e/1b9ddcad-339e-4e4e-84dc-2ec85495d945/role_assignments/daefdb31745f40e5b0ae363f8faefc02>' config={'fetcher': None, 'headers': {'Authorization': 'Bearer permit_key_jLOUM2YHArJsk9twIvJ7xZxMC5GDgIV1ZuPr2N7tBCGT40bzOzW5WErMfBhHMCzSfre42xwLtrlxCsRSiXg5g3', 'X-Permit-Update-Id': '82623316502644c0b49b4732ee1c8c5c'}, 'is_json': True, 'process_data': True, 'method': 'get', 'data': None} topics=['01241a01a63345eebeac1215f8d9d5de:data:policy_data', '01241a01a63345eebeac1215f8d9d5de:data:policy_data/01241a01a63345eebeac1215f8d9d5de'] dst_path='role_assignments/user:68e64179-0082-4665-93a3-0cf9489a5780' save_method='PUT' data=None with exception

Traceback (most recent call last):

  File "/usr/local/lib/python3.10/asyncio/locks.py", line 214, in wait

    await fut

asyncio.exceptions.CancelledError


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for

    return fut.result()

asyncio.exceptions.CancelledError


The above exception was the direct cause of the following exception:


Traceback (most recent call last):

  File "/usr/local/lib/python3.10/site-packages/opal_client/data/updater.py", line 585, in _fetch_data

    result = await self._data_fetcher.handle_url(

  File "/usr/local/lib/python3.10/site-packages/opal_client/data/fetcher.py", line 75, in handle_url

    response = await self._engine.handle_url(url, config=config)

  File "/usr/local/lib/python3.10/site-packages/opal_common/fetcher/engine/fetching_engine.py", line 116, in handle_url

    await asyncio.wait_for(wait_event.wait(), timeout)

  File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for

    raise exceptions.TimeoutError() from exc

asyncio.exceptions.TimeoutError

2026-06-17T22:15:33.715755+0000 | opal_client...base_policy_store_client  |ERROR  | OPA transaction failed, transaction id=82623316502644c0b49b4732ee1c8c5c, actions=[], error=Failed to fetch data for entry <https://pdp-deltas.api.permit.io/v2/internal/opal_data/9f83afe1-55e7-49ee-950e-cac413a75f58/6b3e6334-7228-4b2c-bfff-850caf37027e/1b9ddcad-339e-4e4e-84dc-2ec85495d945/role_assignments/daefdb31745f40e5b0ae363f8faefc02>:

2026-06-17T22:15:33.716750+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://pdp-deltas.api.permit.io/v2/internal/opal_data/9f83afe1-55e7-49ee-950e-cac413a75f58/6b3e6334-7228-4b2c-bfff-850caf37027e/1b9ddcad-339e-4e4e-84dc-2ec85495d945/role_assignments/daefdb31745f40e5b0ae363f8faefc02>

2026-06-17T22:15:33.717630+0000 | opal_client.callbacks.reporter          | INFO  | Reporting the update to requested callbacks

2026-06-17T22:15:33.718059+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://pdp-deltas.api.permit.io/v2/pdps/me/opal_data_callback>

2026-06-17T22:15:33.718550+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: <https://opal-relay.api.permit.io/v2/pdp/callback?retry=0>

2026-06-17T22:15:36.645493+0000 | opal_client.engine.logger               | INFO  | Received request.    GET /health

2026-06-17T22:15:36.646198+0000 | opal_client.engine.logger               | INFO  | Sent response.       GET /health -> 200

2026-06-17T22:15:37.292353+0000 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='<https://pdp-deltas.api.permit.io/v2/internal/opal_data/9f83afe1-55e7-49ee-950e-cac413a75f58/6b3e6334-7228-4b2c-bfff-850caf37027e/1b9ddcad-339e-4e4e-84dc-2ec85495d945/users/daefdb31745f40e5b0ae363f8faefc02>', destination path='/users/68e64179-0082-4665-93a3-0cf9489a5780'

The errors you're seeing are related to OPAL data fetching timeouts during tenant updates. Let me break down what's happening: ​*Key Errors in Your Logs:*​ 1. ​*`asyncio.exceptions.TimeoutError`*​ when fetching role assignments from

<http://pdp-deltas.api.permit.io|pdp-deltas.api.permit.io>

2. ​*`HTTP 503 Service Unavailable`*​ when uploading decision logs 3. ​*`OPA transaction failed`*​ due to the timeout ​*Root Cause:*​ The OPAL client is timing out while fetching role assignment data during tenant updates. The default timeout for OPAL data fetching operations is controlled by

OPAL_FETCHING_CALLBACK_TIMEOUT

. (1) When OPAL receives a data update event (like tenant creation/deletion) , it attempts to fetch the updated data from the control plane. If this fetch takes longer than the configured timeout, the transaction fails. ​*How OPAL Handles Update Events:*​ Once connected, the OPAL client tracks data update events and processes them in order, even before getting the first base data. The client does not re-run update messages after building initial state, but rather spins tasks to update as events come and processes them once possible. ​*Solutions:*​ ​*1. Increase OPAL Fetching Timeout:*​ Add this to your PDP environment variables:

OPAL_FETCHING_CALLBACK_TIMEOUT=60

The default is typically 30 seconds. Increasing it to 60 seconds gives more time for data fetching operations to complete during tenant updates. ​*2. Verify Your Configuration:*​ Your current configuration shows the control plane is overriding

OPAL_FETCHING_CALLBACK_TIMEOUT

from the cloud. The log shows:

Overriden config key: OPAL_FETCHING_CALLBACK_TIMEOUT

This means Permit.io's control plane is setting this value. The timeout might be too low for your use case. ​*3. Monitor for Retry Behavior:*​ The logs show OPAL automatically retries failed fetches. Notice how after the timeout error at

22:15:33.712686

, it immediately tries again at

22:15:33.716750

. Eventually, the data is saved successfully at

22:15:37.292353

. This retry mechanism is built into OPAL - it will continue to try and fetch data until it succeeds (with exponential backoff) . ​*Why This Happens During Tenant Updates:*​ When you create or delete a tenant, multiple data updates are triggered: • User data updates • Role assignment updates • Tenant relationship updates These updates can arrive in quick succession, and if the control plane is under load or network latency is high, individual fetch operations may timeout. ​*The Good News:*​ Despite the timeout errors, your logs show the system eventually recovers and successfully saves the data. The health check at

22:15:36

returns 200, and the user data is saved at

22:15:37

. ​*To help debug this further, please provide:*​ 1. Which Workspace / Project / Env you are using 2. The frequency of tenant creation/deletion operations 3. Whether permission checks fail during these timeout periods 4. Your current

OPAL_FETCHING_CALLBACK_TIMEOUT

value (check via

docker exec <container> env | grep OPAL_FETCHING

) The timeouts appear to be transient and self-recovering, but if they're causing permission check failures, we should investigate further.

To learn more, see the sources I considered: [[1] Data updates](https://docs.opal.ac/faq/)