Hi all Any idea on this session timeout error ```2022 06 13 Apache Pinot #general

Hi all, Any idea on this session timeout error? ``...

Vuppala Suresh Kumar

06/13/2022, 7:10 AM

Hi all, Any idea on this session timeout error?

Copy code

2022-06-13 11:58:16.182  INFO 42149 --- [<http://onaws.com:2181)|onaws.com:2181)>] org.apache.zookeeper.ClientCnxn          : Client session timed out, have not heard from server in 10011ms for sessionid 0x0, closing socket connection and attempting reconnect
2022-06-13 11:58:16.283  INFO 42149 --- [<http://onaws.com:2181)|onaws.com:2181)>] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server <http://ec2-54-214-97-234.us-west-2.compute.amazonaws.com/54.214.97.234:2181|ec2-54-214-97-234.us-west-2.compute.amazonaws.com/54.214.97.234:2181>. Will not attempt to authenticate using SASL (unknown error)

Getting this error while connecting to zookeeper via Java Client

Copy code

Connection connection = ConnectionFactory.fromZookeeper
  ("<xxxxxxxxxxxxx>");

Mayank

06/13/2022, 7:18 AM

Any errors in the server log? It could be GCing

Vuppala Suresh Kumar

06/13/2022, 10:43 AM

Mayank

06/13/2022, 11:30 AM

Oh, this is from the pinot-java-client. Is your ZK running with SASL?

Vuppala Suresh Kumar

06/13/2022, 12:51 PM

yes

Mayank

06/13/2022, 2:44 PM

@saurabh dubey ^^

saurabh dubey

06/13/2022, 4:05 PM

Looking into this

Vuppala Suresh Kumar

06/14/2022, 4:44 AM

@saurabh dubey Got any idea?

saurabh dubey

06/14/2022, 4:48 AM

Could you share you zookeeper.properties file @Vuppala Suresh Kumar?

Vuppala Suresh Kumar

06/14/2022, 4:50 AM

From where we can get that?

saurabh dubey

06/14/2022, 4:50 AM

Did you setup your own zookeper? Or did you use pinot quickStart command?

saurabh dubey

06/14/2022, 4:51 AM

If you could login to any of your zookeeper nodes, it should be present in config/zookeeper.properties

Vuppala Suresh Kumar

06/14/2022, 12:43 PM

Copy code

containers:
- command:
  - /bin/bash
  - -xec
  - /config-scripts/run
  env:
  - name: ZK_REPLICAS
    value: '3'
  - name: JMXAUTH
    value: 'false'
  - name: JMXDISABLE
    value: 'false'
  - name: JMXPORT
    value: '1099'
  - name: JMXSSL
    value: 'false'
  - name: ZK_HEAP_SIZE
    value: 256M
  - name: ZK_SYNC_LIMIT
    value: '10'
  - name: ZK_TICK_TIME
    value: '2000'
  - name: ZOO_AUTOPURGE_PURGEINTERVAL
    value: '0'
  - name: ZOO_AUTOPURGE_SNAPRETAINCOUNT
    value: '3'
  - name: ZOO_INIT_LIMIT
    value: '5'
  - name: ZOO_MAX_CLIENT_CNXNS
    value: '60'
  - name: ZOO_PORT
    value: '2181'
  - name: ZOO_STANDALONE_ENABLED
    value: 'false'
  - name: ZOO_TICK_TIME
    value: '2000'
  image: zookeeper:3.5.5
  imagePullPolicy: IfNotPresent
  livenessProbe:
    exec:
      command:
      - sh
      - /config-scripts/ok
    failureThreshold: 2
    initialDelaySeconds: 20
    periodSeconds: 30
    successThreshold: 1
    timeoutSeconds: 5
  name: zookeeper
  ports:
  - containerPort: 2181
    name: client
    protocol: TCP
  - containerPort: 3888
    name: election
    protocol: TCP
  - containerPort: 2888
    name: server
    protocol: TCP
  readinessProbe:
    exec:
      command:
      - sh
      - /config-scripts/ready
    failureThreshold: 2
    initialDelaySeconds: 20
    periodSeconds: 30
    successThreshold: 1
    timeoutSeconds: 5
  resources:
    limits:
      cpu: 500m
      memory: 1Gi
    requests:
      cpu: 200m
      memory: 256Mi
  terminationMessagePath: /dev/termination-log
  terminationMessagePolicy: File
  volumeMounts:
  - mountPath: /data
    name: data
  - mountPath: /config-scripts
    name: config
  - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
    name: default-token-bsnbh
    readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: shipment-dev-zookeeper-2
imagePullSecrets:
- name: docker-registry-registrycreds
- name: docker-registry
nodeName: ip-192-168-135-4.us-west-2.compute.internal
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
  fsGroup: 1000
  runAsUser: 1000
serviceAccount: default
serviceAccountName: default
subdomain: shipment-dev-zookeeper-headless
terminationGracePeriodSeconds: 1800
tolerations:
- effect: NoExecute
  key: <http://node.kubernetes.io/not-ready|node.kubernetes.io/not-ready>
  operator: Exists
  tolerationSeconds: 300
- effect: NoExecute
  key: <http://node.kubernetes.io/unreachable|node.kubernetes.io/unreachable>
  operator: Exists
  tolerationSeconds: 300
volumes:
- name: data
  persistentVolumeClaim:
    claimName: data-shipment-dev-zookeeper-2
- configMap:
    defaultMode: 365
    name: shipment-dev-zookeeper
  name: config
- name: default-token-bsnbh
  secret:
    defaultMode: 420
    secretName: default-token-bsnbh

saurabh dubey

06/15/2022, 6:40 AM

Are you able to connect to this zookeeper cluster using java zookeper client?

saurabh dubey

06/15/2022, 8:22 AM

Infact, the fact that your pinot cluster is up using the same Zookeper, means SASL is not being used by the helix client. Did you have to provide any JAAS conf files to controller, broker and server? When starting the pinot server?

Vuppala Suresh Kumar

06/16/2022, 5:56 AM

2022-06-16 112222.946 INFO 64752 --- [ main] org.apache.zookeeper.ZooKeeper : Initiating client connection, connectString=pinot-zookeeper-dev.fareye.ml sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@222acad 2022-06-16 112251.697 INFO 64752 --- [onaws.com:2181)] org.apache.zookeeper.ClientCnxn : Opening socket connection to server ec2-52-88-237-233.us-west-2.compute.amazonaws.com/52.88.237.233:2181. Will not attempt to authenticate using SASL (unknown error) 2022-06-16 112557.260 INFO 64752 --- [onaws.com:2181)] org.apache.zookeeper.ClientCnxn : Client session timed out, have not heard from server in 211974ms for sessionid 0x0, closing socket connection and attempting reconnect

Vuppala Suresh Kumar

06/16/2022, 7:21 AM

Any idea on Null pointer exception at org.apache.pinot.client.ExternalViewReader.unpackZnodeIfNecessary(ExternalViewReader.java:129) ~[pinot-java-client-0.9.2.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]

Copy code

2022-06-16 12:46:45.484  INFO 73824 --- [           main] org.apache.zookeeper.ZooKeeper           : Initiating client connection, connectString=localhost:2191 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@51f68849
2022-06-16 12:46:45.498  INFO 73824 --- [localhost:2191)] org.apache.zookeeper.ClientCnxn          : Opening socket connection to server localhost/127.0.0.1:2191. Will not attempt to authenticate using SASL (unknown error)
2022-06-16 12:46:45.504  INFO 73824 --- [localhost:2191)] org.apache.zookeeper.ClientCnxn          : Socket connection established to localhost/127.0.0.1:2191, initiating session
2022-06-16 12:46:45.512  INFO 73824 --- [localhost:2191)] org.apache.zookeeper.ClientCnxn          : Session establishment complete on server localhost/127.0.0.1:2191, sessionid = 0x100027658760001, negotiated timeout = 30000
2022-06-16 12:46:45.534  WARN 73824 --- [           main] o.a.pinot.client.ExternalViewReader      : Exception while reading External view from zookeeper

java.lang.NullPointerException: null
	at org.apache.pinot.client.ExternalViewReader.unpackZnodeIfNecessary(ExternalViewReader.java:129) ~[pinot-java-client-0.9.2.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
	at org.apache.pinot.client.ExternalViewReader.getTableToBrokersMap(ExternalViewReader.java:94) ~[pinot-java-client-0.9.2.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
	at org.apache.pinot.client.DynamicBrokerSelector.refresh(DynamicBrokerSelector.java:70) ~[pinot-java-client-0.9.2.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
	at org.apache.pinot.client.DynamicBrokerSelector.<init>(DynamicBrokerSelector.java:56) ~[pinot-java-client-0.9.2.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
	at org.apache.pinot.client.ConnectionFactory.fromZookeeper(ConnectionFactory.java:55) ~[pinot-java-client-0.9.2.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
	at org.apache.pinot.client.ConnectionFactory.fromZookeeper(ConnectionFactory.java:43) ~[pinot-java-client-0.9.2.jar:0.9.2-3e5e5aca07ad796d453add80a2b2b1bf1d654cf1]
	at com.fareye.shipment.visibility.layer.data.config.PinotConfig.getConnection(PinotConfig.java:24) ~[main/:na]
	at com.fareye.shipment.visibility.layer.data.config.PinotConfig$$EnhancerBySpringCGLIB$$ae8bbe55.CGLIB$getConnection$0(<generated>) ~[main/:na]
	at com.fareye.shipment.visibility.layer.data.config.PinotConfig$$EnhancerBySpringCGLIB$$ae8bbe55$$FastClassBySpringCGLIB$$d2df33a2.invoke(<generated>) ~[main/:na]
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.3.10.jar:5.3.10]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.3.10.jar:5.3.10]
	at com.fareye.shipment.visibility.layer.data.config.PinotConfig$$EnhancerBySpringCGLIB$$ae8bbe55.getConnection(<generated>) ~[main/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1380) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1300) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[spring-beans-5.3.10.jar:5.3.10]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.10.jar:5.3.10]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.10.jar:5.3.10]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.5.5.jar:2.5.5]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-2.5.5.jar:2.5.5]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) ~[spring-boot-2.5.5.jar:2.5.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) ~[spring-boot-2.5.5.jar:2.5.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-2.5.5.jar:2.5.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) ~[spring-boot-2.5.5.jar:2.5.5]
	at com.fareye.shipment.visibility.app.ShipmentVisibilityApplication.main(ShipmentVisibilityApplication.java:11) ~[main/:na]

2022-06-16 12:46:46.854  INFO 73824 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8100 (http) with context path ''
2022-06-16 12:46:47.375  INFO 73824 --- [           main] c.f.s.v.a.ShipmentVisibilityApplication  : Started ShipmentVisibilityApplication in 60.191 seconds (JVM running for 60.567)

saurabh dubey

06/16/2022, 7:28 AM

@Kartik Khare https://github.com/apache/pinot/blob/master/pinot-clients/pinot-java-client/src/main/java/org/apache/pinot/client/ExternalViewReader.java#L75 We're trying to read

Copy code

byte[] brokerResourceNodeData = _zkClient.readData(BROKER_EXTERNAL_VIEW_PATH, true);

But ideally we should be reading

Copy code

byte[] brokerResourceNodeData = _zkClient.readData("/<clusteName>" + BROKER_EXTERNAL_VIEW_PATH, true);

saurabh dubey

06/16/2022, 7:52 AM

@Vuppala Suresh Kumar You should be putting your clusterName in the zkurl being used. i.e.

Copy code

Connection connection = ConnectionFactory.fromZookeeper
  ("localhost:2191/PinotCluster");

or whatever your cluster name is. Could you try with this and see if it works?

Open in Slack

Previous Next