Hey all. I have a question relating to using Pulsar as a KV store (for a chat system). Is this possible? Does this negate the need for having consumers? Are they needed at all? Especially in a replicated environment. Would only 1 consumer need to exist to acknowledge the memory and mark it as a stream? Confused on this point. Also what are the ramifications. Is it possible to use it to store day 2 days worth of messages and then to slowly migrate older messages to a sink and have this as a continual process? Finally, how would querying the data occur in this process? Would it be possible to always query the KV store and if older messages are requested. Does the KV store request the data from the sink database? Or does this logic have to exist in the query logic? To pull from both the KV store AND sink database and merge both streams and return the data back to the client? Just trying to get my head around the process. Many thanks!

Hi Paul, it’s hard to give a general answer but to give you some starting points, the Pulsar concepts you want to look into are topic compaction and possibly using a TableView as a queryable KV store. Continuously sinking records is possible using the IO connectors. How all of these pieces fit together exactly will depend on your system design

@Alexander Preuß Thanks for the reply. Appreciate it! I’ve been thinking about this all day and due to the complexity have decided to drop it. Your reply was the nudge I needed. Many thanks!