https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
l
Have you tried druid-operator to run the cluster in kubernetes? It supports to upscale/downscale services under utilization metrics. Example
Copy code
hpAutoscaler:
        maxReplicas: 10
        minReplicas: 1
        scaleTargetRef:
           apiVersion: apps/v1
           kind: StatefulSet
           name: druid-tiny-cluster-brokers
        metrics:
         - type: Resource
           resource:
             name: cpu
             target:
               type: Utilization
               averageUtilization: 50
on this case everytime k8s will spawn new broker pods (up to 
maxReplicas
= 10) when cpu usage of current ones is > 50%. When the load is down, it shuts down no longer needed instances (but never less than 
minReplicas
amount). And if you using k8s you can even use middlemanager-less cluster, so it spins small pods when you have ingestions and they die as soon as the task is done.
j
I'm afraid we're not using kubernetes, just regular docker; switching is definitely on the table but not our case yet
thanks for the info!
a
@Jordi Escrich @Luiz Augusto If you are running on k8s, you can run without MM https://druid.apache.org/docs/latest/development/extensions-contrib/k8s-jobs.html
l
@Adheip Singh I mentioned this in my last message ☝️
g
in general, tasks aren't kept around longer than they need to be — druid will shut them down when their work is done
is there a reason you need to shut them down early in step (2b)? (instead of wait for them to exit)
j
hey, sorry @Gian Merlino I missed your message. so what I'm doing is shutting down the middlemanager containers themselves once they are idle (0 tasks running out of X slots)
apparently that messes running tasks up due to later stages somehow using data from middlemanagers that participated in other earlier stages of that particular ingestion task
I verified that by checking the payloads of tasks failing for apparently no reason, and indeed they were referencing middlemanagers that I killed due to them being idle
so I took the naive approach of just waiting for all tasks to be completed to scale middlemanagers down (effectively killing the containers); we currently only have need for them during a relatively small chunk of time during the day, so it's kind of working for now
but I was kind of surprised about this unexpected "coupling" between specific nodes in between stages
g
ahhh, I see. Yes, MMs provide a shuffle service that is used for 
index_parallel
subtasks; the data on the shuffle service outlives the task themselves (but does not outlive the 
index_parallel
task)
with SQL-based ingestion using MSQ we redesigned the shuffle mesh to not use the MM service, so it's been decoupled. You can be sure that if the tasks are gone then the MM is no longer needed. You may find value in switching to MSQ for this reason (and others! it's more robust generally in other ways too)
Another option, if you'd like to stick with `index_parallel`: you can use 
druid.processing.intermediaryData.storage.type = deepstore
to use remote storage (S3, etc) as the shuffle service for 
index_parallel
. That would also make it safe to terminate the MMs once the tasks are gone
j
oh! both outstanding suggestions Gian, thank you very much!! I’ll go with deep storage first as the quick and easy solution, and definitely eventually migrate to MSQ (have been looking forward to it); thanks again!
2 Views
Open in Slack
PreviousNext
Powered by