Hi @Yosi Haran do you have any update on timing for this? I believe @Mike Hudson was going to check back today so apologies if it's a duplicate request.

@Yosi Haran checking back on this. Any update?

Hi @Yosi Haran do you have an update on timing for this week?

@Yosi Haran Below are the logs

Hey Vivek, I took a look at the duration reported in the logs, and the longest performance I saw was

208ms

, during a cache miss, which is reasonable run time for a network request, and I see most requests taking under

1ms

, which means they are returning directly from the local cache. Perhaps there is an external issue related to the Prisma database connection which might be configured differently when authzcache is used? Maybe threads are not being closed properly? The failures I saw in the logs did not seem related to authzcache, but to Postgress. After my manual analysis, I ran an analysis using Claude Code and verified that it's conclusion about authzcache logs, which I am familiar with, is correct. I cannot verify the conclusions about the service logs, but perhaps it will be useful for you? Please see the analysis below and let me know if it makes sense 🙏

authzcache performance (from dev-auth-cache-logs.csv)

  Extracting all gRPC durations from the Check calls:

  ┌──────────┬────────┬──────────┐
  │   Time   │ Method │ Duration │
  ├──────────┼────────┼──────────┤
  │ 07:54:06 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:54:26 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:54:26 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:54:29 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:54:29 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:54:48 │ Check  │ 33ms     │
  ├──────────┼────────┼──────────┤
  │ 07:54:50 │ Check  │ 200ms    │
  ├──────────┼────────┼──────────┤
  │ 07:54:50 │ Check  │ 208ms    │
  ├──────────┼────────┼──────────┤
  │ 07:54:51 │ Check  │ 23ms     │
  ├──────────┼────────┼──────────┤
  │ 07:55:06 │ Check  │ 132ms    │
  ├──────────┼────────┼──────────┤
  │ 07:55:26 │ Check  │ 18ms     │
  ├──────────┼────────┼──────────┤
  │ 07:55:26 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:55:28 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:55:29 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:55:49 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:55:49 │ Check  │ 1ms      │
  ├──────────┼────────┼──────────┤
  │ 07:55:50 │ Check  │ 5ms      │
  └──────────┴────────┴──────────┘

  The vast majority of Check calls complete in 1-5ms. The occasional 100-200ms calls are likely cache misses that proxy to the remote Descope server — entirely normal behavior. Write operations (CreateFGARelations, DeleteFGARelations) also
   complete quickly at 29-34ms.

  The "idle-session timeout" errors (from dev-service-action-logs.csv)

  The "terminating connection due to idle-session timeout" errors at lines 364-383 are PostgreSQL errors (SqlState E57P05), NOT authzcache errors. They come from the DatabaseService / Prisma client within the actions service itself:

  prisma:error: Error in PostgreSQL connection: Error {
    severity: "FATAL",
    code: SqlState(E57P05),
    message: "terminating connection due to idle-session timeout"
  }

  These occur at 07:58:00-07:58:01 (6 errors) and 07:59:50 (2 errors). Their Azure PostgreSQL server (aptly-pg-server-dev.postgres.database.azure.com) is killing idle connections in Prisma's connection pool.

  Later at 08:13:29, the service restarts and fails entirely with P1001: Can't reach database server — the PostgreSQL server is completely unreachable.

  What's actually happening

  1. The client's PostgreSQL has idle_session_timeout configured — it kills connections that sit idle in Prisma's pool. When the service tries to use a killed connection, it gets the FATAL error.
  2. Their database becomes entirely unreachable at ~08:13, causing bootstrap failures and crash loops.
  3. They're checking permissions for 1,206 actions per request (visible in multiple log entries like "Checking permissions for 1206 actions"). Even at 1-5ms per authzcache call, that adds up — but it's the sheer volume, not authzcache
  latency.

  Verdict

  authzcache is fast and healthy — 1-5ms typical response times. The latency/timeout issues are entirely on the client's infrastructure side:

  - Root cause: Their Azure PostgreSQL server's idle_session_timeout setting is killing pooled connections, and the database is intermittently unreachable.
  - Recommended actions for the client:
    a. Check/increase the idle_session_timeout setting on their Azure PostgreSQL server
    b. Configure Prisma's connection pool to handle stale connections (e.g., connection_limit, pool_timeout, or connection health checks)
    c. Investigate why their database becomes completely unreachable (network/firewall/Azure scaling issue)

@Vivek Sharma Please see above, forgot to tag you in the original msg 🙂🙏

So, it's looking normal... can you perhaps provide a bit more data or client-side logs which show extended latency when using authzcache? 🙏

I am also continuing further analysis of the authzcache to make sure I didn't miss anything and will update if I see anything new...

Also, perhaps you can provide a larger log window for the authzcache? I double-verified that I didn't miss any long running request in the logs, e.g. a request that started but didn't report duration, but all "starts" are showing an "end" matching with the same RID, and all latencies look good

@Yosi Haran Please find the attached logs. Let me know anything else is required from my end

I can see all 29 requests have matching start+end, and durations are looking good, summary listed below. Nothing pointing at higher latency in the updated logs either, I'm afraid... Perhaps you can point to to the specific service-side log which indicates that latency is higher?

┌──────────────┬───────┬─────────────────────────────────────────┐
  │   Duration   │ Count │                 Details                 │
  ├──────────────┼───────┼─────────────────────────────────────────┤
  │ 0ms (sub-ms) │ 2     │ 109µs, 112µs — quick cache hits.        │
  ├──────────────┼───────┼─────────────────────────────────────────┤
  │ 1-2ms        │ 14    │ Normal cache hits                       │
  ├──────────────┼───────┼─────────────────────────────────────────┤
  │ 25-32ms      │ 5     │ Write ops (Create/Delete FGA) + 1 miss  │
  ├──────────────┼───────┼─────────────────────────────────────────┤
  │ 108-186ms    │ 5     │ (Larger) Cache misses proxied to remote │
  └──────────────┴───────┴─────────────────────────────────────────┘

I am consulting with @Eliya Sadan, we think it might actually be a too aggressive timeout on the Node-SDK side, causing the FGA cache call to timeout too quickly and then fallback to a direct call, which will cause unnecessary added latency, We've released a version to address this which should be available shortly, I will let you know once it's available so you can update the SDK 🙏 Additionally, we added more analytics to the authzcache docker image, which I will share an updated version of soon. This will not resolve the issue, but will help us view aggregated performance data for checks more easily.

@Yosi Haran Unfortunately still the same result. Attaching logs for your reference

@Vivek Sharma or can you please share (or tag Harsh to share) the actual value of the FGA cache URL you are using in the SDK? Also, do you have any other services calling authzcache? I'm afraid that the CORS issue is a red herring related to the gateway timeout response. Perhaps the issue is that the specific requests sent from the client we looked at together are not reaching the authzcache, due to some issue with the FGA Cache URL being used in that specific service, and the logs we are seeing there are from other usages in the code?

Another data point which points to request likely not reaching authzcache is that the default timeout there is 10 seconds, so a failure would have been returned sooner if the requests were reaching it

Good morning/afternoon everyone, just updating that I'm not sure yet but might have an angle with the node-sdk, will have a fix ready for testing shortly. Thank you again for your patience and clear cooperation with this issue, it's very much appreciated 🙏

@Yosi Haran Good news, its working. Next, I need to compare response times with and without fga cache for 1000+ records. cc - @Mike Hudson @Kenneth Knoll

(and as described in the dockerhub link, you may also opt-out of this by adjusting the env variables, although only aggregated performance statistics are collected without any PII of course)

@Harsh Gajjar we fixed the build issue and released a new version which should work on both arm and amd, can you please pull the latest image again and let me know if the issue was resolved? 🙏

@Eliya Sadan FYI ☝️🙏

Hey @Vivek Sharma, I analyzed the logs and collected the aggregated metrics from our end. I want to be sure I'm looking at the correct time window, since I'm seeing numbers that are a bit different, but could be because of a simple time window difference. I think we can already see that cache hits reduce the waiting time to 1ms at most, which is good, but perhaps we can further optimize our backend performance for the bulk-check use case... I'd just like to make sure my understanding is correct, so please let me know when would be a good time for a short 15 min zoom and I'll be more than happy to send an invite 🙏

@Vivek Sharma one more question, to confirm: can you please share the descope projectId that you were using for these tests? 🙏

Hi @Vivek Sharma, good morning, the performance fix is live, feel free to re-run the test when ready If you can also share the updated results I'd really appreciate it, and as always please feel free to keep me posted and lmk if there are any other issues 🙏

@Vivek Sharma the deploy looks good, but I'm having trouble finding calls from your projectID in our backend, I see 0 calls for

P2mDFvFZYGZ4pt1KzQVgaF89Dmr9

🤔 Could you please re-confirm and let me know which projectId you used? 🙏

I managed to find the relevant data. The query that best matches the long running query below arrived and completed within

4ms

. Additionally, the cache metrics do not show any long running queries that match the

~8s

delay we are seeing in the logs below... However, I see that only 16 tuples were checked in the backend, and 80 were requested from the cache. So I will dig into the cache logic to make sure we do not have any potential issue there either, and will update more soon. @Vivek Sharma From your end, can you please verify that

setupDescopeClient()

does not cause any locking or waiting for other requests to finish? Seems unlikely, but I don't know what this method does, so worth verifying as well if that's ok 🙏

public async checkRelationFGA(relations: IFGARelation[]): Promise<IFGACheckMap[]> {
    await this.setupDescopeClient();  // <-- THIS
    const response = await this.descopeClient.management.fga.check(relations);
    return response.data ?? [];
}

{"timestamp":"2026-04-16T09:01:50.130Z","status":"debug","level":"debug","message":"[checkPermissions] Calling Descope checkRelationFGA API","service":"Actions service","env":"development","logger.name":"ActionService"}
{"timestamp":"2026-04-16T09:01:58.744Z","status":"debug","level":"debug","message":"[getGlobalSearchActionList] transformActionToResponseDto branch completed in 8613ms","service":"Actions service","env":"development","logger.name":"ActionService"}
{"timestamp":"2026-04-16T09:01:58.874Z","status":"debug","level":"debug","message":"[checkPermissions] Descope checkRelationFGA API completed in 8744ms (response size: 80 checks)","service":"Actions service","env":"development","logger.name":"ActionService"}

If you see permissionBranchPromise here it calls a checkPermissions function which do not include any local DB calls. It just creates relations and then check those in Descope