Yeah, I don't want to disturb the upper thread, so I'll start a new one, but the question is quite related. How should we work with

zedtoken

in a microservice architecture when permissions are granted/revoked by admins? For granting permissions, there is no big problem. You can grant the permission in one service and eventually send the

zedtoken

to another service or store it in a DB. It is acceptable if the user gets access a couple of seconds later. But what happens when we remove/revoke a permission? Revoking a permission should ideally take effect immediately. Imagine an admin revokes a user's permission in one service/pod. We then need to propagate the corresponding

zedtoken

to the service handling the user's queries as quickly as possible. If everything is handled within one microservice, we have the problem of storing the

zedtoken

in two databases (SpiceDB and our DB). Potentially, we could lose the latest

zedtoken

if our DB transactions fail. If the permission management and user queries are handled by different microservices, we have the same persistence problem, plus eventual consistency. We need to somehow transfer the

zedtoken

to another microservice, which could take some time depending on the implementation (Kafka, etc.). So, in this case, understanding whether a relation was deleted by an admin is not immediately possible unless we use

fully_consistent

for such queries. Am I missing something here? Is there a recommended pattern for handling

zedtoken

propagation in this kind of architecture, especially when permission revocation needs to take effect immediately? Thanks in advance.

is your system synchronous or eventually consistent? how do you currently handle dual writes when a request spans services?

Yeah, thanks for the answer. It depends. Part of our system is synchronous, but another is eventually consistent. We solve dual writes in a different way, but mostly with the outbox pattern.

that's similar to what we did at my old job

Thanks! What you wrote is correct, and I don't have any objections. What I'm trying to understand is this. Imagine your system is under attack and someone's JWT tokens have been stolen. You need to revoke that user's permissions immediately. With eventual consistency, you'll eventually propagate the new ZedToken. But imagine the outbox is down, or it's heavily delayed because of an attack or extreme load. In that case, the service performing authorization checks won't know about the latest ZedToken during read operations. It will have to wait until the ZedToken is delivered, or until the datastore-revision-quantization-interval expires before it starts seeing the updated permissions. How would you handle this scenario?

are you referring to the new enemy problem?

> are you referring to the new enemy problem? > like the idea that a user shouldn't be able to see anything post-revocation? Yeah, sorry for confusing, I am probably about this one. Performance of SpiceDB is not a question. The question is how to immediately see revoked permissions if we are using ZedToken.

so one of the keys here is that if someone's permission is revoked, but they're using a zedtoken that lets them see a cached result, they aren't seeing anything new

Yeah, for sharing photos if you already see them or sharing some documents if you saw them, it is probably okay. But, for example, if you restrict admin ability to approve withdrawals, it is a bit of a different kind of problem.

right

Yes, that's exactly the question. It seems that it's technically impossible to propagate a ZedToken without some delay, so the only solution for these security-critical cases is to use

full_consistency

. No matter which pattern you choose, you end up with the same fundamental problem: you're writing to two different data stores. You first call

DeleteRelation

in SpiceDB and receive a ZedToken, and then you somehow need to persist or propagate that token. Whether you use an outbox, Kafka, or any other approach, there will always be a window where the token hasn't been stored or delivered yet, because those operations can't be performed atomically.

yeah, that's true. we have kicked around the idea of a zedtoken jar internally, where there's a layer of abstraction that lets you refer to a zedtoken via an ID or set of IDs, and changes that involve those IDs bump a cached revision such that the next request that references those IDs (by ID, rather than by zedtoken) picks up the most recent revision

Yes, I think so. Many thanks, @yetitwo! This conversation helped clarify things for me, and I now feel confident that I wasn't missing any key concepts on this topic. Thanks again for taking the time to answer my questions! Need to think a bit about this 🙂

sure thing!