This message was deleted.
# generals
Slackbot
09/28/2023, 9:02 AMThis message was deleted.
b
Bibek Sahoo
09/28/2023, 9:39 AMPlease go through below discussion where the reason and alternative approach is discussed.
https://apachedruidworkspace.slack.com/archives/C0303FDCZEZ/p1695102588220579
s
Satakshi Pal
09/28/2023, 2:30 PMokay thanks
j
John Kowtko
09/30/2023, 4:27 PMInsert and replace are batch operations in Druid ... and since Druid segments are immutable the only way to get new data into the system is to create new segments.
This is what auto-compaction was designed to help with ... if you are running batch ingestions then you can set up your auto-compaction offset to PT0S so it will run against the latest time chunk whenever you don't have a batch ingest also running at the same time.
Best practices? In your case it might be to put Kafka in the middle and have your API calls instead push data into Kafka, and let Druid pull out of Kafka via streaming ingestion ... this will likely produce far fewer segments.