https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
🙌 1
g
Yes, it's a number of bytes
k
thanks. but then 300,000,000 bytes = 300Mb while the default calculates to 1Gb? How does providing less space works?
g
Hmm. It shouldn't… Do you have 
-Xmx16g
in your historical 
jvm.config
and 
druid.processing.numMergeBuffers = 4
in your historical 
runtime.properties
?
k
yes
g
hmm let me see if the same thing happens on my machine
I tried those settings and the merging dictionary size was indeed set to 1GB by default
So I'm not sure what's going on
Maybe the settings aren't taking, somehow? Possible that the service is not actually getting launched with 
-Xmx16g
and 
druid.processing.numMergeBuffers = 4
?
k
yes, from jvm file
Copy code
-Xms2g
-Xmx16g
-XX:MaxDirectMemorySize=14g
and historical/runtime.properties
Copy code
druid.server.http.numThreads=60
druid.processing.buffer.sizeBytes=500MiB
druid.processing.numMergeBuffers=4
druid.processing.numThreads=15
g
hmm. strange that i do not see the same behavior. is this something that can be reproduced using druid 25.0.0 from druid.apache.org? if so, please let us know how to repro it and we will figure it out!
k
How do I figure out the merging dictionary size for my cluster?
Also, http//<druid broker>8082/status returns { "version": "0.22.1", "modules": ... Seems i on a primitive version
i
is the xms coming into play? it starts at 2g
if the JVM never has to increase it may be coming in smaller than the 1GB? Just a thought
v
In Druid 25 there is a new `start-druid` script that auto sizes all the services memory configs is that something that alleviates this issue?
k
"Druid 25" - I am afraid, I cannot check that since my company uses the older version. "is the xms coming into play" - yes, thats a good point. I am trying to find ways to see how much memory is currently with the dictionary. is there any way to check that?
i
you could just try setting xms and xmx the same and see if the behavior persists
just wonder if it is looking at the startup heap and determining the size then and not adjusting as the heap grows which would place it lower than 1GB
k
Yes, I will try setting Xms and Xmx as same. Current /status from Historiical
Copy code
"memory": {
   "maxMemory": 15271460864, ~15G Xmx
   "totalMemory": 4903141376, ~4.9G
   "freeMemory": 3141649328,  ~3G
   "usedMemory": 1761492048,  ~1.7G
   "directMemory": 15032385536~15G
}
seems like only 1.7G is being used and so if the dictionary mem is calculated as per this, it will indeed be way less.
Obs on setting Xms = Xmx, mem usage:
Copy code
"memory": {
  "maxMemory": 17015767040,		17G
  "totalMemory": 17015767040,	17G
  "freeMemory": 15408552704,	15G
  "usedMemory": 1607214336,		1.6G
  "directMemory": 15032385536	15G
}
Still the usage is 1.6G, so I guess I will have to setup "_maxMergingDictionarySize_" via config, which is kind of dicy. i should't set it to 300Mb but to 1Gb then. which will be the max it will ever get - MIN(30%heap, 1Gb)
i
hmmm the jvm should have started with the 16GB if they were both set to that?
s
Could it be being calculated off of totalMemory? 4.9 * 0.3 / 4 = 0.367 but I guess that would still be higher than 300MiB...
g
Ah, I just noticed you pasted something saying your Druid version is 0.22.1
In that version, merging dictionary was hard-coded to 100000000 (i.e. 100MB)
🫡 1
The adaptive dictionary size based on available memory was added in a later version
So, I bet that's what's going on
You should upgrade 🙂 but if you can't, then yeah you should manually set the dictionary size
👍 1
k
Thank you all for your input. this is very valuable. I couln't get this info from anywhere. 🙏
g
that's why we're here in slack 🙂
Open in Slack
PreviousNext
Powered by