https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
b
small point - stay under 32G. Even 32765M. At 32G, internal pointers go from 32-bit to 64-bit, something about that, it's worse unless you get up to somewhere like 40G. 31G or 32765M might be better. I didn't find the best reference, but you can google 
java heap 32G
. :)
y
Thank you 🙂
g
note that in Druid 25 (the latest release) you can start druid via the 
bin/start-druid
script and it will automatically determine reasonable configs given the available resources on the server. i would start from that as a baseline
y
Will try. Thanks
Cause now, I really have no luck with the Indexers 😞
And it may just be an issue with k8s/jvm
a
@Gian Merlino 
bin/start-druid
how does the script work ? will auto-bootstrap configs ? Just thinking from k8s side, we still need add in resources depending upon the druid nodetype etc
g
It looks at the number of processors and amount of memory on the machine, and which set of services are run; then sets the basic configs (like Xmx, direct memory, etc) to reasonable defaults
You can also provide a specific number of processors and memory if you don't want to use everything available on the machine
a
Copy code
k logs -f druid-e2e-cluster-brokers-748ddd54d9-z8pxz  -n druid
python interepreter not found
seems like it will need an init-container to bootstrap python
which latest druid has this supported ?
y
Finally figured out why the Indexers were failing! maps! Had to set
Copy code
vm.max_map_count=1048576
Indexer is generating a lot of intermediate files during ingestion!
g
That thread is about direct buffer memory, but merging phase can also use a lot of mmaps
TLDR is that if you use SQL-based batch ingest (via MSQ tasks) then it is tuned automatically and you don't have to worry about it
If you are using other forms of ingest, you can adjust that if you have any further issues
y
We're planning for MSQ
g
👍
2 Views
Open in Slack
PreviousNext
Powered by