we are running cluster data, master, broker nodes,...
# general
s
we are running cluster data, master, broker nodes, running into frequent crashes for broker, overlord. off late zookeeper seems to be unresponsive due connection. ui doesnt show up the segments and just get hung. overlord crashes 3-4 times a day. how to improve on this
g
is anything interesting being logged?
if "everything" is crashing for no apparent reason it may be you are running out of memory
s
we have 2Million segments. It makes zookeeper crash 1st and then overlord, historicals. post restart it works. but due to ingestion, compaction and its constantly phenomenon
post crossing 1.6 millions it became bottleneck for metadata postgres, zookeeper and overlord crashing due to supervisors running
memory utilisation of zookeeper jvm is 4GB+, historical is 120GB+, brokers 90GB+ per nodes
we used to have same no .of segments loading in all historical nodes. but recently i see different count of segments loading one of node than the others
g
sorry, I missed responding for a while. 2 million is quite a lot of segments. it is possible to tune the system to work well with 2 million segments, but before going down that route, I'd want to first confirm if you actually need 2 million segments. what is their average size, in number of rows? generally we look to have 3–5 million rows per segment on average. if yours are smaller than that, then I would recommend using compaction to reduce the number of segments. on the other hand, if yours are about that size, that means you have a trillions of rows dataset. congrats, that's really big data šŸ˜‰. in this case i have a few things to consider: • what version of Druid are you using? with more recent versions, ZooKeeper is used for much less- pretty much just leader election and service discovery. There should be no ZooKeeper scalability issues with large segment counts in the more recent versions • for the Broker and Overlord both, I'm curious what kinds of crashes are you seeing? You mentioned having 90GB+ for Brokers- even at 2 million segments that should be more than enough memory. I wonder if you are seeing OutOfMemoryErrors or something else