Hello team, I have a question about Kafka and Pinot integration. Is there a way to store all Kafka partitions within a single Pinot segment? Also, could you explain the reasoning behind the current setup where it seems to be tied to: 1 Kafka partition -> 1 Pinot consumer -> 1 segment? What are the implications or benefits of this design? Thanks!

Could you elaborate why you would like to have all Kafka partitions into one segment? Once segments are generated you can combine/ merge / rollup them using minions. One consumer per partition is very scalable

f.e. I start with low load on start and my events are partitioned by tenant_id in kafka topic so increasing number of partitions is hard -> good to start from right size of partitions for future load( f.e. 50) So i see right approach would be to merge segments to efficient size after they are flushed to disk?

You can always increase partitions later on and Pinot will automatically ingest from them without any human interference needed