https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
s
Is that config from the druid operator? I'm not very familiar with it, but it seems to me that you are creating 12 middle manager pods and that each one will spawn up to 12 peons (druid_worker_capacity). The middle manager itself doesn't need a lot of memory, so you could reduce that. Each Peon within each MM pod will be using 5GB for heap and up to 10GB of direct memory (druid_indexer_running_javaOpts). So your pods should be configured for at least 13 cpus and memory 13*15GB = 195 GB. If your pod request limits are smaller it is likely that it will cause 137s.
j
It is from druid helm chart not the operator. Where did the 13 came from? 15GB I understand that is heap plus direct memory
however resource limits for MM looks like:
Copy code
resources:                                                                                                                                                                                                                                    
       limits:                                                                                                                                                                                                                                     
         cpu: "32"                                                                                                                                                                                                                                 
         memory: 48Gi                                                                                                                                                                                                                              
       requests:                                                                                                                                                                                                                                   
         cpu: "32"                                                                                                                                                                                                                                 
         memory: 48Gi
so that might be the problem
the instance types I am using for MM are 
c5d.9xlarge
however I think I might need to use 
i3.8xlarge
instead as they offer up tp 244GB ram in comparison to 72GB for the c5d
s
Yeah, bigger nodes would help or you can reduce the memory settings and test your ingestion workload to see how much you can do with the c5d.9xlarge nodes. It is 13 because I am counting the MM JVM as well as the 12 worker JVMs. You can probably also reduce the settings for the MM heap and directmemory.
j
the problem I guess too are realtime queries going on together with realtime ingestion, so that memory consumption will increase
s
I'm trying to understand all the memory footprint of real-time ingestion tasks as it relates to their ingestion and query activities. On the query side, in principle, you have a controlled set of threads : 
druid.processing.numThreads
that each use a set of merge buffers 
druid.processing.numMergeBuffers
of size 
druid.processing.buffer.sizeBytes
and all this is off-heap memory. The HTTP threads for the task are used to receive requests and functions as a queue, the requests are processed from this queue using processing threads to resolve individual queries. I'm still not sure about what this activity does on heap, but I'm digging deeper 😉
j
all right, thanks. Yeah the whole thing is why I started getting tasks killed with code 137 and all I thought of was probably memory allocation
however what you described with regards to params, seems to be more related to historical nodes, and I would relate that to historical type of queries. Tasks being exited with 137 code belong to middleManagers I think
and I think the problem is with MM because of resource usage across all MM, which seems to be over 90% in all 12 MM I have, sometimes 99% in some of them
s
Yes, The query side of the ingestion tasks is similar to what Historicals do.
But it works on the intermediate persists and the in-memory row ingestion buffer instead of working on a set of segments.
How big are a your ingestion? Messages/second ? How many queries/second on real-time?
j
I would say over 90k messages per second ingested, in spikes perhaps over 120k. Whereas queries received, over last month roughly 200k queries
Not sure if this count if what you look for
both metrics from January 2023
s
I'm just trying to get a sense of the volumes and concurrency. Have you seen this doc? It has a section "A few numbers" on results from running different message schemas and the ingestion rate they achieved per task. It can serve as a guide to what you might expect from your ingestion.
j
Haven't seen it yet. Thanks for it
s
How did it go? What settings did you end up with? Were you able to test up to 120k messages per second? Given that you have swings in the message rate, it may also make sense for you to look at the lag based autoscaler which will increase and decrease the task count dynamically based on the consumption lag.
j
With the instance change to 
i3.8xlarge
the problem got resolved, not more failed tasks with 137 and mem usage seems normal now. We are using Kinesis, is that autoscaler applicable to kinesis ingestion too? I was thinking in a similar approach but by monitoring lag metrics emitted to statsd and via datadog
s
Kinesis ingestion also supports it. One thing to consider is that you will need to have enough middle manager/indexer worker slots available. An interesting test for this would be to also use 
kubernetes-overlord-extensions
to run without middle managers on k8s. Not sure how that will behave because it is still experimental, but it promises to provide more elasticity with workers since it spawns workers as k8s jobs, no MM, the Peons are spawned as pods. Not sure how adventurous you are feeling 😄
j
I might test that out in the dev cluster tho and see how it goes. For a production environment tho, it might be risky if it’s still experimental
s
agreed
Open in Slack
PreviousNext
Powered by