Hi team, I was looking to see. Is there any specific difference in processing times between doing a compaction job, where we change a data source from hour granularity to day granularity, this is for a hashed partitioned data set, in contrast to using an index job re-ingesting it from a druid dataSource as input to a new dataSource. For reference we are looking to test reducing from an hourly granularity to daily granularity as currently we are storing 560 segments per day for 26gb of data and the high count of segments appears to be significantly impacting the ability for caching to succeed and subsequently making queries slower. From testing re-ingesting a days data from a druid dataSource its taking much too long to do, i'm wondering if its different at all from doing it in a compaction job. We don't want to impact the current data in the table by running a compaction job so I'm limited in my capability to test this currently.

A compaction job also does re-ingestion under the hood so there shouldn't be any difference in terms of processing time.

The only significant "ingestion-time" difference I can think of would be in the parallelization that you set up for the job. Can you share your job spec so we can take a look?

Hi @John Kowtko here is the spec we have access to deploy our own middlemanager cluster for this. I'm wondering if scaling up the workers and increasing max rows in memory could speed things up by reducing the flushing?

Thanks for sharing the spec. You have

maxNumConcurrentSubtasks

already set high, to 50, that's good. You should also set

totalNumMergeTasks

to match or exceed the number of segments you anticipate the job producing, so that these tasks can run in parallel as well. And yes check your available task slots to make sure subtasks aren't stuck in Pending mode waiting for slots to become available. Can you check one of your log files to find the log statements for "estimatedNumSucceededTasks" and let us know what number it is giving? This is where for each phase the controlling task will determine how many subtasks it will execute ... then it is a question of whether or not those subtasks will execute in parallel or not. Then you want to make sure you have

maxNumConcurrentSubtasks

,

totalNumMergeTasks

, and enough task slots to allow these phases to all run in parallel. I also noted that your maxSplitSize is set to 100MB ... the default I think is 1GB ... if your estimatedNumSucceededTasks is very high for the first phase of the reindex job then maybe your setting is too low. It would probably be helpful if you shared a full log file (from the parent task) so we could look for other issues.