Hi folks, trying out datahub on gcp and got thinking. What is Kafka used for exactly ? More importantly I would like things to be easy so doing a thought experiment about replacing it with pubsub or better yet a interface so that it's swappable. Anyone else has thoughts on this?

Hi @calm-airplane-47634: Kafka is used as the metadata log to capture all changes to metadata and route them to the indexes (search, graph). It is definitely (logically) possible to replace it with another pub-sub system. We don’t have this on our short-term roadmap though.