Felix GV
09/05/2023, 12:19 AM5. During the last sync session, you mentioned that a push job creates a new topic each time it runs. If we understood that correctly, what's the motivation for it?Yes, that is correct. There are a few motivations for this design choice… 1. Each dataset version having its own "version topic" simplifies the follower replica and Da Vinci code paths: rehydration requires just scanning the topic of the current version from beginning to end. 2. Also, this allows us to delete the old version topic to be retired (thus reclaiming space faster), rather than relying on time retention (which provides much less control). 3. Finally, in a hybrid (batch+stream) store, the outcome of merging a streaming write on top of whatever was there before could lead to a different outcome if the value that was pushed from batch is different. This complexity is encapsulated in the leader replica, which writes to its version topic. Hope that makes sense…!