This message was deleted.
# troubleshootings
Slackbot
11/16/2022, 5:01 PMThis message was deleted.
s
Sergio Ferragut
11/16/2022, 6:03 PMHi @sridhara. Do you mean you are attempting to use zookeeper as the metadata database for Druid? I don't think that is a supported configuration. Zookeeper is used in Druid for interprocess communication and for leader election, but I've never seen it used as a Metadata database component. For that you can use postgres, mysql. Or am I misunderstanding?
s
sridhara
11/16/2022, 6:19 PM@Sergio Ferragut yeah it is for interprocess communication onIy... what metadata i mean is here the segments information we store right that is why i just mentioned as metadata
s
Sergio Ferragut
11/16/2022, 6:23 PMI see. Okay, can you provide more context? How are you deploying it? Is this a fresh install? Does this occur on startup or when does it occur?
s
sridhara
11/16/2022, 6:27 PM@Sergio Ferragut It was working for months now and all of sudden we started seeing this issue today... where in our zookeeper went into bad state and we lost quorum... This is the error that zookeeper is failing with
Copy code
Unable to load database on disk
java.io.IOException: Unreasonable length = 8522996sridhara
11/16/2022, 6:28 PMwe tried to restore from the backup it was functioning for 2 hours and then it started the issue again
sridhara
11/16/2022, 6:29 PMsridhara
11/16/2022, 6:30 PMwe are trying to understand is above issue was due to above jira
k
Karan Kumar
11/16/2022, 6:41 PMI am not sure what this bug is but I would try to use zookeeper as less as possible.
For that use http based load queues and http task runner. I think that would elevate the pressure on zookeeper.
s
Sergio Ferragut
11/16/2022, 6:54 PMJust to add to @Karan Kumar’s point. Those settings are:
druid.serverview.type=httpdruid.coordinator.loadqueuepeon.type=httpdruid.indexer.runner.type=httpRemotes
sridhara
11/16/2022, 8:24 PM@Arvind Singh 👆
a
Arvind Singh
11/16/2022, 8:29 PMWe are getting the above issue and zookeeper is not coming up. Here Zookeeper is running as a Kubernetes service.
Arvind Singh
11/16/2022, 8:31 PMIs there anyway to bring up zookeeper by resolving the below issue:
Copy code
2022-11-16 20:18:54,446 [myid:3] - INFO [main:ZKAuditProvider@42] - ZooKeeper audit is disabled.
2022-11-16 20:18:54,606 [myid:3] - ERROR [main:QuorumPeer@1148] - Unable to load database on disk
java.io.IOException: Unreasonable length = 8523180d
Didip Kerabat
11/16/2022, 8:34 PMBy any chance, is your ingestion spec huge?
a
Arvind Singh
11/16/2022, 8:35 PMYes
d
Didip Kerabat
11/16/2022, 8:35 PMWe had this same issue before. The huge payload is bumping against ZK’s default limit.
Didip Kerabat
11/16/2022, 8:36 PMDo you deploy ZK with Bitnami Zookeeper Helm chart?
Didip Kerabat
11/16/2022, 8:39 PMStep 1: Enlarge the limit on ZK itself:
Copy code
jvmFlags: "-Djute.maxbuffer=16777216" Copy code
-Djute.maxbuffer=16777216🔥 1
a
Arvind Singh
11/16/2022, 8:40 PMThanks a lot. Let me try it
s
sridhara
12/21/2022, 7:28 AM@Didip Kerabat Should we keep the max jute buffer size for zookeeper and druid same ?
If it is same will that not effect zookeeper anything?
d
Didip Kerabat
12/21/2022, 3:52 PMyes but had to be the same
10 Views