Hello I have operational questions about Datahub inner workings, especially with respect to the Kafka topics used internally. At this moment my team and I are running Datahub in production with all Kafka topics required by the project (https://datahubproject.io/docs/how/kafka-config/#topic-configuration) configured with a single partition - we were extremely early adopters of Datahub and in the beginning we were worried about ordering of events if topics were partitioned - which of course comes at a price when performance is considered. Therefor we have below questions: 1. Can we use partitions for all (or part) of the topics used by Datahub? Are there any experiences as to what numbers should be used here? 2. How many parallel consumers should be used depending on the partitions of the topics? Does it make sense to have dedicated consumers for separate topics (metadata change log vs metadata change proposal) or is it better (performance wise) to have each consumer consume all of them? 3. As we are running highly-available service and pull metadata as well as receive metadata via pushes we can not stop ingestion of the metadata. If we wanted to recreate topics with higher partition-count we planned to simply create new topic, change deployment of consumers/gms to point at new topic and roll them out one at a time (to keep HA), drain old topic into new one (to get metadata which was received during transition period) - in this case there is possibility that older events will be processed after newer ones - will they be properly treated (discarded) and not override newer data?

Hello Piotr, 1.) Yes, you can use partitions for topics. I have personally used 3 partitions for all topics by default and then more than that for highly used topics like MCP & MCL topics which partitions up to 16. The number of partitions highly depends on your use-case and the level of horizontal scaling you require per consumer. 2.) This is standard Kafka so you should use 1 consumer per partition for the most processing speed. This requires more resources for your deployment. It’s a balancing act, if you are ok with 1 consumer handling multiple partitions. I.e: Kafka lag in a topic not increasing then I would suggest having 1 consumer deal with as many partitions as possible to reduce operational costs. DataHub uses dedicated consumer groups for each topic. No single consumer handles messages from different topics. I would suggest not changing this. 3.) DataHub merges the data from your metadata events into a single document in it’s various data stores (ElasticSearch & SQL respectively). If they are disjoint you are ok. By disjoint I mean that message M1 and M2 update different sections of an entity or aspect. If they modify the same thing then the last to arrive wins. Note however, that since you have Kafka topics you can work with retention time to serve as a buffer of work to be processed. This means you could use kafka connect or something similar to move your events from the old topic to the new one. See this tutorial from confluent: https://developer.confluent.io/tutorials/change-topic-partitions-replicas/ksql.html

Hi @incalculable-ocean-74010 thanks for the answer, but it seems to me there is no eventual consistency in L-DH. Did I understand correctly that if I have a dataset with Status(removed=False) and then I receive MCPs (with their respective timings): 1. Event 1 - Status(removed=True) at 12:05 2. Event 2 - Status(removed=False) at 12:06 And then these events end up in 2 different partitions there is a chance one consumer will process Event 2 first and then Event 1 which would mean the dataset would be softly deleted (this logic would apply to any other aspect, here used just for an example). I thought if partitions are allowed then timing of event generation would be taken into consideration and Event 1 would be discarded during processing. It is not true I understand?

We have eventual consistency once the messages are in kafka. Before that no. We do not use event time to handle out-of-order events.