This message was deleted.

If you are referring to editing the data within a segment -- segments in Druid are immutable (cannot be modified), but there are ways to edit data within segments by reingesting modified data on top of the existing segment, in whole or in part. If you are referring to changing the datasource structure to have segments of different granularity -- that is done through reindexing or compaction. Information on both of the above can be found in the docs: • https://druid.apache.org/docs/latest/data-management/update.html • https://druid.apache.org/docs/latest/data-management/compaction.html What specifically would you like to do?

thanks specifically speaking I want to edit the configuration of segments of my existing table so that from now on the segments are formed properly.. now the segments are being formed haphazardly and it is causing issue while running queries

Ingestion will generally result in segment fragmentation, i.e. multiple segments for the same time chunk, often smaller than ideal size. This is due to the way the parallel ingestion processes work, and also the fact that the data coming in can have __time values spanning multiple time chunks. Compaction generally cleans all that up. The simplest way to get compaction going is to turn on auto-compaction on the datasource. That will give you a good first pass at cleaning up the segment fragmentation due to ingestion. There are other steps you can take beyond that to tune the system for better storage optimization and query performance, but just setting up default auto-compaction is usually a first good step.

Hello John, I have 1 question to learn, Doing compaction will reduce the performance of Querying data from an application & Ingestion ? Example case, If I have 10 data-sources and each have approx., 5 million records/rows already with 'n' number of segments, will it work without disturbing the other Queries / Ingestions ? (am using single-server configuration, even-though if we increase the worker-capacity i.e. service slots) Thanks to Viraj ! Thanks for his question !

It is set up to operate with a limited set of resources (something like 10% of the task slots by default.) It also can have locking contention with ingestion, but there is an "offset" you can set so that compaction starts with older time chunks that are not currently being ingested into. So for example if you get data in that has timestamps throughout the day, then a common setting for auto-compaction is to have a "P1D" offset so it will only compact yesterday's and older data. There are still a few caveats here based on what data is coming in -- read up on lateMessageRejectionPeriod and earlyMessageRejectionPeriod parameters ... this is perhaps one of the reasons why this is not enabled by default. That being said, there are some enhancements in the works to all but eliminate lock contention between compaction and ingestion.

Thanks @John Kowtko I wasn't aware bout this segment fragmentation, can you shed some light on this , I didn't reall understood what exactly is segment fragmentation so later on I can understand about auto compaction that you mentioned

https://druid.apache.org/docs/latest/data-management/compaction.html : Each peon or worker task on Middle Managers will generate and publish segments async to deep storage. Ideally, each segment should reach 5M or 300-700MB but it’s not always the case. Hence the need for compaction