Hi, I am seeing 2 segments in consuming state for my table and rest are online. Is that expected behavior? Also the consumption is very slow after a certain number of records 3M. What could be the reason for this?

Number of segments in

CONSUMING

state should be the same as number of partitions.

Ok

If consumption is slow, there can be multiple reasons. Given enough cpu and memory pinot will consume as fast as the producer produces data. So, if you see a mismatch, it is either cpu or memory on the consumer side. Too many partitions in a single host? Too much paging going on? A burst of queries stole cpu from consuming threads?

I think I also remember seeing a discussion recently where there was an issue upstream in the kafka broker(s). So you may want to check on that as well.