Luis Fernandez
05/12/2022, 2:12 PMMayank
Luis Fernandez
05/12/2022, 2:20 PMMayank
Luis Fernandez
05/12/2022, 2:24 PMMayank
Daniel Lavoie
05/12/2022, 2:31 PMLuis Fernandez
05/12/2022, 2:36 PMDaniel Lavoie
05/12/2022, 2:39 PMkubectl logs pinot-zookeeper-1 --previous
Luis Fernandez
05/12/2022, 2:41 PM2022-05-12 10:15:32,603 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@430] - Have smaller server identifier, so dropping the connection: (3, 2)
2022-05-12 10:15:32,604 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:32,614 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x110001d0cf (n.zxid), 0x6 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x12 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:32,615 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x110001d0cf (n.zxid), 0x6 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x12 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:32,816 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 400
2022-05-12 10:15:32,816 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@430] - Have smaller server identifier, so dropping the connection: (3, 2)
2022-05-12 10:15:32,817 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:32,817 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x110001d0cf (n.zxid), 0x6 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x12 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:33,217 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 800
2022-05-12 10:15:33,221 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@430] - Have smaller server identifier, so dropping the connection: (3, 2)
2022-05-12 10:15:33,221 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:33,221 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x110001d0cf (n.zxid), 0x6 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x12 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:34,022 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 1600
2022-05-12 10:15:34,023 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@430] - Have smaller server identifier, so dropping the connection: (3, 2)
2022-05-12 10:15:34,023 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:34,023 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x110001d0cf (n.zxid), 0x6 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x12 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:35,624 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 3200
2022-05-12 10:15:35,624 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@430] - Have smaller server identifier, so dropping the connection: (3, 2)
2022-05-12 10:15:35,624 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:35,625 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x110001d0cf (n.zxid), 0x6 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x12 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:38,825 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 6400
2022-05-12 10:15:38,826 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@430] - Have smaller server identifier, so dropping the connection: (3, 2)
2022-05-12 10:15:38,826 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:38,826 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x110001d0cf (n.zxid), 0x6 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x12 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:45,227 [myid:2] - INFO [QuorumPeer[myid=2](plain=/0.0.0.0:2181)(secure=disabled):FastLeaderElection@919] - Notification time out: 12800
2022-05-12 10:15:45,228 [myid:2] - INFO [WorkerSender[myid=2]:QuorumCnxManager@430] - Have smaller server identifier, so dropping the connection: (3, 2)
2022-05-12 10:15:45,228 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 2 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:45,233 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@679] - Notification: 2 (message format version), 3 (n.leader), 0x110001d0cf (n.zxid), 0x6 (n.round), FOLLOWING (n.state), 1 (n.sid), 0x12 (n.peerEPoch), LOOKING (my state)0 (n.config version)
2022-05-12 10:15:56,491 [myid:2] - INFO [NIOWorkerThread-1:FourLetterCommands@234] - The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}]
2022-05-12 10:15:56,491 [myid:2] - INFO [NIOWorkerThread-1:FourLetterCommands@235] - The list of enabled four letter word commands is : [[wchs, stat, wchp, dirs, stmk, conf, ruok, mntr, srvr, wchc, envi, srst, isro, dump, gtmk, telnet close, crst, cons]]
2022-05-12 10:15:56,491 [myid:2] - INFO [NIOWorkerThread-1:NIOServerCnxn@518] - Processing ruok command from /127.0.0.1:56218
2022-05-12 10:15:57,298 [myid:2] - INFO [NIOWorkerThread-2:NIOServerCnxn@518] - Processing srvr command from /127.0.0.1:56224
Daniel Lavoie
05/12/2022, 2:42 PMkubectl describe pod pinot-zookeeper-1
Should give an exit status for previous containerLuis Fernandez
05/12/2022, 2:42 PMLast State: Terminated
Reason: Error
Exit Code: 143
Daniel Lavoie
05/12/2022, 2:43 PMMayank
Daniel Lavoie
05/12/2022, 2:45 PMLuis Fernandez
05/12/2022, 2:47 PMDaniel Lavoie
05/12/2022, 2:48 PMLuis Fernandez
05/12/2022, 2:49 PMkubectl describe pod pinot-zookeeper-1
Daniel Lavoie
05/12/2022, 2:49 PMkubectl describe pod pinot-zookeeper-1 | grep OOM
Luis Fernandez
05/12/2022, 2:50 PMDaniel Lavoie
05/12/2022, 2:50 PMLuis Fernandez
05/12/2022, 2:54 PMresources: {
requests: {
cpu: '500m',
memory: '1Gi',
},
limits: {
cpu: '500m',
memory: '1Gi',
},
},
ZK_HEAP_SIZE: '256M',
Daniel Lavoie
05/12/2022, 2:55 PMLuis Fernandez
05/12/2022, 3:00 PM2022-05-06 15:55:56,927 [myid:3] - WARN [LearnerHandler-/10.24.10.144:59094:LearnerHandler@928] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:926)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2022-05-06 15:55:56,927 [myid:3] - WARN [QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled):ZooKeeperThread@55] - Exception occurred from thread QuorumPeer[myid=3](plain=/0.0.0.0:2181)(secure=disabled)
java.lang.OutOfMemoryError: Java heap space
2022-05-06 15:55:56,927 [myid:3] - WARN [LearnerHandler-/10.24.7.181:38132:LearnerHandler@928] - Ignoring unexpected exception
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:339)
at org.apache.zookeeper.server.quorum.LearnerHandler.shutdown(LearnerHandler.java:926)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:647)
2022-05-06 15:55:56,928 [myid:3] - INFO [main:QuorumPeerMain@104] - Exiting normally
pinot-zookeeper-2
Daniel Lavoie
05/12/2022, 3:01 PMLuis Fernandez
05/12/2022, 3:09 PMDaniel Lavoie
05/12/2022, 3:10 PMLuis Fernandez
05/12/2022, 3:23 PMDaniel Lavoie
05/12/2022, 3:24 PMLuis Fernandez
05/12/2022, 3:24 PMMayank
Luis Fernandez
05/12/2022, 7:04 PMDaniel Lavoie
05/12/2022, 7:20 PMLuis Fernandez
05/12/2022, 7:27 PMDaniel Lavoie
05/12/2022, 7:45 PMFizza Abid
06/28/2022, 11:27 AM