Hi Druids, when using a real time ingestion using kafka, using multiple MiddleManagers, where are located the fresh data? Do they stick to one segment in one MM, are they balanced between all MM, or are they replicated on all MM?

Hi Julien, The ingestion tasks are mostly what matter here ... All partitions in a stream are round-robin assigned to the ingestion tasks running under a given supervisor. An ingestion task can have one or more Kafka partitions assigned to it. So a given Kafka partition is consumed by only one ingestion task by default, when ingestion task replicas are not being used. The tasks are generally just spread out across MMs based on available task slots. If you are running with ingestion replication, meaning more than one replica, then each ingestion task has replica tasks which ingestion the same set of partitions, and they all do identical work in parallel. In this case the supervisor must spread the replicas for a given ingestion task to run on different MMs. In all of the cases above, the data stays on the MM node, either in memory or temporarily on local disk, where in both stages it is considered a "real-time" segment.

It makes sense, thank you very much.