This message was deleted.
# generals
Slackbot
09/08/2023, 3:04 PMThis message was deleted.
j
John Kowtko
09/08/2023, 3:36 PMHi Guillaume,
Compaction is limited to using only a portion of the worker task slots available on the nodes ... I think the default is 10%. There is a parameter that allows you to adjust it ... looking for it now ...
g
Guillaume Leclanche
09/08/2023, 3:36 PMI have configured those parameters
j
John Kowtko
09/08/2023, 3:37 PMIt should be able to run multiple compactions on the same datasource (different time chunks) simultaneously though ...
g
Guillaume Leclanche
09/08/2023, 3:37 PMyes it is
Guillaume Leclanche
09/08/2023, 3:37 PMbut not "in between"
Guillaume Leclanche
09/08/2023, 3:37 PMlet's say I have 5 compaction tasks started at 06:00 because there are enough slots available
Guillaume Leclanche
09/08/2023, 3:38 PMat 07:00, one completes, at 08:00 another one, etc.
Guillaume Leclanche
09/08/2023, 3:38 PMit looks like there won't be any new compaction task started until ALL 5 are completed
j
John Kowtko
09/08/2023, 3:38 PMI thought they were independent ... i.e. when a slot opens, a new job should be able to use it.
g
Guillaume Leclanche
09/08/2023, 3:39 PMwhile I would expect to have all the time roughly 5 compact tasks running all the time
Guillaume Leclanche
09/08/2023, 3:39 PMmodulo "indexingperiod", which in our case is 10m
j
John Kowtko
09/08/2023, 3:48 PMcompactionTaskSlotRatiog
Guillaume Leclanche
09/08/2023, 3:50 PM"compactionTaskSlotRatio" : 0.6,
"maxCompactionTaskSlots" : 225
yep it's tuned like that on our side
j
John Kowtko
09/08/2023, 3:52 PMThen possible the subTasks are taking up the task slots.
It is stated somewhere that once a task starts, if it needs more subtasks than there are slots available, it will still launch the subtasks.
So for a while you will be oversubscribed on task slot usage. which means when some of the jobs complete, you still may be consuming all of the available task slots with the subtasks of the remaining jobs running.
Can you monitor the "Services" tab to check the available task slots during this time?
g
Guillaume Leclanche
09/08/2023, 3:53 PMthey are freed properly
Guillaume Leclanche
09/08/2023, 3:56 PMThis is the status right now
s
Suneet Saldanha
09/08/2023, 4:49 PMwhich version of Druid are you running? Starting in Druid 23, you can configure the compact segments duty to run more frequently than other indexing duties - https://github.com/apache/druid/pull/12263
Suneet Saldanha
09/08/2023, 4:50 PMThis will make it so that Druid checks for available capacity for compaction more frequently.
Suneet Saldanha
09/08/2023, 4:53 PMDruid does not necessarily know the actual capacity needed for a compaction task, so it assumes each compact task will take the
maxNumConcurrentSubtasksg
Guillaume Leclanche
09/13/2023, 8:44 AMYep the concept is clear, and we've tuned the parameter accordingly. It's really about the synchronous start of all tasks. Since they don't have the same duration, when the fastest compact task is completed, a new one should be started, and slowly they should all no longer be in sync. But I do see them always with the same start date, which makes me think that no new compaction tasks are started for a datasource as long as at least one is still running. And that doesn't seem right/efficient.
Guillaume Leclanche
09/13/2023, 8:48 AMAnyway I'll open a github ticket for that
j
John Kowtko
09/13/2023, 9:41 AMHi Guillaume, others can confirm, but I believe the "Created time" for tasks is the time when the entry is added to the task table, not when the task actually starts ... for ingestion for example, the task will actually start when a worker slot opens up. We've seen this with ingestion task cycling for taskDuration ... so if you are making assumptions as to when tasks are starting -- or getting held up from starting -- based on those Created dates, that might be misleading.