Hi all, we're running some operations on spicedb in a script which is lightly hitting the service to set up some relationships. We run this script and everything goes well until we randomly get "403 unauthorized" errors when calling spicedb. This happens on read and write calls, I don't see a consistent set of records it's choking up on. Any idea what the issue may be or where I should investigate?

this is OSS SpiceDB? are you using the HTTP API or the gRPC API?

This would be OSS and using the gRPC endpoints, we have a preshared key associated with the 2 instances we have. Right now our best lead is we see a pg connection error that the remaining slots are depleted although spicedb is set to 30 connections for each instance so 60 out of our 100 connections

and the SpiceDB call itself is returning 403 unauthorized?

I'm not entirely certain yet. I haven't looked into the load balancer or whatever else is in the way in google cloud when hitting the domain but the errors are coming from the client call to the spicedb rpc in my Go code. We're running on a simple VM and not the operator in k8s

yeah, it's weird to me that it's at the level of http