This message was deleted.
# troubleshootings
Slackbot
11/25/2022, 5:30 PMThis message was deleted.
r
Renato Santos
11/27/2022, 1:10 PMs
Sean F
11/28/2022, 3:16 PMthanks! my point was that it seemed that mm tasks upload and announce segments even before task duration is exhausted (for example when segment granularity or rows per segment is met). my question was, in that case, what is really the significance of task duration? or do I misunderstand something?
r
Renato Santos
11/28/2022, 11:48 PMI think it's mainly related to the segments merging, but I need to check that with empirical testing (because I can't read the java code to follow the 'path'), but I think if you set task duration 1hr, but limit each segment to 1 row (that would be very inefficient) and during the task duration, you would have multiples segments, but after the task finishes¹ it should merge then all back into N optimized segments, even if you don't have any auto-compaction enabled
¹ not really finished, but after the new offsets are passed into another MM task be the owner
Renato Santos
11/28/2022, 11:52 PMhmm
so, according to the docs, this merge does not happen automatically, so task duration is either to you control planned downtime, but that could also be archived by intermediateHandoffPeriod, so now I'm confused 😆
Renato Santos
11/29/2022, 12:02 AMafter reading https://druid.apache.org/docs/latest/design/architecture.html I'm certain that after the handoff, the task offsets are updated as well, so the MM task may only be useful for metrics accumulation?!
Renato Santos
11/29/2022, 12:06 AMI guess I was confused about
Intermediate persist periodIntermediate handoff periodg
Gian Merlino
11/30/2022, 8:11 AMkafka tasks run for a
taskDurationGian Merlino
11/30/2022, 8:12 AMduring that time, they may do intermediate handoffs, where they publish some data due to reaching a trigger like a certain number of rows
Gian Merlino
11/30/2022, 8:12 AMor, they may not, if the data volume isn't super high
Gian Merlino
11/30/2022, 8:12 AMeither way, when they reach the
taskDurationGian Merlino
11/30/2022, 8:12 AMby this time, they will have been replaced with a new 'wave' of tasks
Gian Merlino
11/30/2022, 8:13 AMthe purpose for the waves (rather than using the same tasks forever) is mainly so task logs don't get too long, and so you can do rolling updates of MMs more easily (by disabling them and then waiting for their tasks to exit)
🙏 1
✅ 2
Gian Merlino
11/30/2022, 8:13 AMhope this clears things up; feel free to ask something else if not 🙂
2 Views