Hello, I'm running pinot on docker, and am creating and pushing new data/ segments (from csv files) into pinot every 1 minute. I realize that the amount of time to push the segment into pinot increases as the data/ segment increases, to the point where it take more than a minute to push the new segment into pinot. How can I make the pushing of a segment faster? Thanks.

Why every minute? If you plan to push that frequently, it’s might be better to stream it in via Kafka

The existing services are using pulsar to stream data, and pinot doesnt have pulsar connector yet, hence the batch ingestion. Around 100kb ~ 3mb per file.

@Kishore G not sure if you can help with this?