Hi, observed that increasing the zk client timeout...
# troubleshootinge
Elon
11/15/2021, 8:19 PMHi, observed that increasing the zk client timeout in the pinot zookeeper does not prevent a zk client timeout from helix, which is hardcoded. We see these errors when the brokers are under heavy gc pressure, gc pauses, etc.
Elon
11/15/2021, 8:20 PM Copy code
org.apache.helix.manager.zk
public class ZkClient extends org.apache.helix.manager.zk.zookeeper.ZkClient implements HelixZkClient {
...
public static final int DEFAULT_SESSION_TIMEOUT = 30 * 1000;Elon
11/15/2021, 8:20 PM@Junkai Xue @Kishore G
Elon
11/15/2021, 8:20 PMWould it make sense to make this configurable?
Elon
11/15/2021, 8:21 PMI do agree w @Kishore G’s concern that the longer the timeout the longer an outage is not detected
m
Mayank
11/15/2021, 8:34 PM
Not to side track the conversation @Elon, but are you seeing 30s GCs?
e
Elon
11/15/2021, 8:35 PMThanksfully we didn't but we did see a spike in gc activity/duration
Elon
11/15/2021, 8:35 PMStill investigating, could be a readiness probe failing
p
Peter Pringle
04/08/2022, 11:47 AMwe have the same issue, we should have a configurable zk timeout
Peter Pringle
04/08/2022, 11:48 AMnot some constant value
k
Kishore G
04/08/2022, 1:17 PM
There is a jvm flag system property to configure it
p
Peter Pringle
04/08/2022, 1:19 PMI couldn't spot it in the code, if you have the variable we can give it a try
k
Kishore G
04/08/2022, 2:01 PM
Copy code
public static final String FLAPPING_TIME_WINDOW = "helixmanager.flappingTimeWindow";
// max disconnect count during the flapping time window to trigger HelixManager flapping handling
public static final String MAX_DISCONNECT_THRESHOLD = "helixmanager.maxDisconnectThreshold";
public static final String ZK_SESSION_TIMEOUT = "zk.session.timeout";
public static final String ZK_CONNECTION_TIMEOUT = "zk.connection.timeout";