Hi All, i have configured REALTIME tables with below configuration "stream.kafka.topic.name": "JOB_summary", "realtime.segment.flush.threshold.segment.rows": "60000000", "realtime.segment.flush.threshold.time": "6h", "stream.kafka.consumer.type": "lowlevel", For some tables, i have below configuration "stream.kafka.topic.name": "pm_datas", "realtime.segment.flush.threshold.rows": "2000000", "realtime.segment.flush.threshold.time": "6h", "stream.kafka.consumer.type": "lowlevel", Query: 1. There is no event ingested in KAFKA in one server. But i am able to see after 6h once there is empty "0" records segments were created. 2. But at the same time, it is not generated for pm_datas file and it is generated only on JOB_summary table. Why there is a empty segment created, even there is no data stream of data in KAFKA? What was the design of pinot and what scenarios other than the above configuration, SEGMENTS were created in deepstore? Thanks SP.

it means there is no data in kafka for pinot to consume or pinot consumer cannot decode the data correctly

That's correct. I don't want or I have not added my kafka ingestion or kafka data published in kafka topic. This is to keep the setup ready with pinot, kafka topic configuration done... I kept the system up and running for 3-4 days. This is done purposefully to check what is happening when there is no KAFKA data in the topics. how does pinot segment creation works. It seems it generates empty segments under ./deepstore directory. Next step, I did a segment upload on certain tables. The task went fine. It is a scheduled job and it creates multiple iterations. post to this jobs also, i do see some "0" rows segements were created. Now the query is why i am getting empty segments?

Can you post the segment Metadata here