https://pinot.apache.org/ logo
#general
Title
# general
j

Jinwei Zhu

01/11/2021, 11:01 PM
Hi @User I'm working with @User and trying to use our new Pinot Kinesis support. Want to know do we have any images built on that? Because with the branch, we can not use it directly. Thanks
n

Neha Pawar

01/11/2021, 11:14 PM
we haven’t built any images. You could build one from the branch: https://docs.pinot.apache.org/operators/tutorials/build-docker-images
j

Jinwei Zhu

01/12/2021, 1:08 AM
Thanks Neha, I tried it according to the doc and got the error
[ERROR] Failed to execute goal com.mycila:license-maven-plugin:2.8:check (default) on project pinot-kinesis: Some files do not have the expected license header -> [Help 1]
[ERROR]
 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] <http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException>
[ERROR]
 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]  mvn <args> -rf :pinot-kinesis
The command '/bin/sh -c git clone ${PINOT_GIT_URL} ${PINOT_BUILD_DIR} &&   cd ${PINOT_BUILD_DIR} &&   git checkout ${PINOT_BRANCH} &&   mvn install package -DskipTests -Pbin-dist -Pbuild-shaded-jar -Dkafka.version=${KAFKA_VERSION} &&   mkdir -p ${PINOT_HOME}/configs &&   mkdir -p ${PINOT_HOME}/data &&   cp -r pinot-distribution/target/apache-pinot-*-bin/apache-pinot-*-bin/* ${PINOT_HOME}/. &&   chmod +x ${PINOT_HOME}/bin/*.sh' returned a non-zero code: 1
do you know how to deal with it?
n

Neha Pawar

01/12/2021, 1:15 AM
looks like some files are missing license headers. I’ll add them and update the branch
thankyou 1
you can try again
j

Jinwei Zhu

01/14/2021, 10:50 PM
Hi Neha, I deployed the new image and seems like the UI of Pinot doesn't running by using the new image. Do you have any ideas with that?
n

Neha Pawar

01/14/2021, 10:52 PM
that is strange. that should not happen.. the kinesis branch is only a few commits behind master, and the UI code was merged long time ago
what exactly do you mean by the UI is not running? what do you see?
j

Jinwei Zhu

01/14/2021, 11:02 PM
The UI is a blank page, when I switch back to the official image, the UI come back gain.
n

Neha Pawar

01/14/2021, 11:03 PM
did you try clearing browser cache ?
j

Jinwei Zhu

01/14/2021, 11:29 PM
Just worked! Thanks Neha, but one more question, why brokers all dead with this image. I redeployed few times but brokers keep dead
n

Neha Pawar

01/14/2021, 11:31 PM
Why do you say they are dead? You’re seeing errors when you try to query? Sometimes the cluster can take a few minutes to be all active
j

Jinwei Zhu

01/14/2021, 11:32 PM
message has been deleted
n

Neha Pawar

01/14/2021, 11:36 PM
hmm, can you check broker logs? are servers alive?
j

Jinwei Zhu

01/14/2021, 11:43 PM
In the broker logs, there is an error like this
Copy code
2021/01/14 17:10:27.612 WARN [ClientCnxn] [Start a Pinot [BROKER]-SendThread(wb-metrics-zookeeper.wb-metrics.svc.cluster.local:2181)] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_275]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_275]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-efbeea2bf8159eb7299292929a57c47345feb341]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-efbeea2bf8159eb7299292929a57c47345feb341]
n

Neha Pawar

01/14/2021, 11:47 PM
so brokers are not able to connect to Zookeeper? Are other components able to start up fine like servers and controller?
@Daniel Lavoie @Xiang Fu could one of you help to investigate this one?
j

Jinwei Zhu

01/14/2021, 11:48 PM
Yes, others are able to start up fine
x

Xiang Fu

01/14/2021, 11:50 PM
hmm can you try to delete the broker pods and see it can fix the issue
this is a warning log should reconnect to zk
j

Jinwei Zhu

01/15/2021, 6:31 PM
Yes, i tried deleting them, when they start again, still shown dead in the UI
x

Xiang Fu

01/15/2021, 6:55 PM
Hmm, any thing found in broker logs
j

Jinwei Zhu

01/15/2021, 6:58 PM
So wired the logs looks good.
Copy code
2021/01/15 18:28:51.181 INFO [HelixBrokerStarter] [Start a Pinot [BROKER]] Starting Pinot broker
2021/01/15 18:28:51.189 INFO [HelixBrokerStarter] [Start a Pinot [BROKER]] Connecting spectator Helix manager
2021/01/15 18:28:52.241 INFO [HelixBrokerStarter] [Start a Pinot [BROKER]] Setting up broker request handler
2021/01/15 18:28:54.144 INFO [HelixBrokerStarter] [Start a Pinot [BROKER]] Starting broker admin application on port: 8099
Jan 15, 2021 6:28:58 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:8099]
Jan 15, 2021 6:28:58 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
x

Xiang Fu

01/15/2021, 7:26 PM
hmm
can you check controller zookeeper browser and see what’s under LIVEINSTANCES?
message has been deleted
are those brokers joined as expected
j

Jinwei Zhu

01/15/2021, 7:32 PM
No, they are not showing there
x

Xiang Fu

01/15/2021, 7:41 PM
hmm, are those broker containers are running ? can you enter into the container and see the pinotBroker.log file
that should give extra logs than kubectl logs
j

Jinwei Zhu

01/15/2021, 8:29 PM
Hi @Neha Pawar I just debugged with Xiang and seems there are some problems in the kinesis branch. Because when switch back to the original image, it works well.
n

Neha Pawar

01/15/2021, 8:32 PM
did you figure out what’s the exact problem? was there nothing in the logs?
@Xiang Fu could some problem have come up because of the jackson version upgrade? that is the only thing in that branch which would affect brokers
x

Xiang Fu

01/15/2021, 8:34 PM
could be
but we don’t see anything from the log
i feel you can try to start controller/broker/server separately
and see if that can be reproduced
n

Neha Pawar

01/15/2021, 9:02 PM
i’m able to start all components separately, locally just fine. I’ve been testing it that way using StartZK, StartController StartBroker and StartServer locally
x

Xiang Fu

01/15/2021, 10:27 PM
hmm
which branch are you using, I can try to build one docker image using that
n

Neha Pawar

01/15/2021, 10:28 PM
sharded_consumer_type_support_with_kinesis
x

Xiang Fu

01/15/2021, 11:10 PM
@Jinwei Zhu can you try to deploy this image:
apachepinot/pinot:sharded_consumer_type_support_with_kinesis
I built it from the branch neha provided
n

Neha Pawar

01/15/2021, 11:13 PM
it worked for you @Xiang Fu ?
j

Jinwei Zhu

01/15/2021, 11:25 PM
Hmm, seems the same as before.😔
Any updates?🙂
n

Neha Pawar

01/19/2021, 8:35 PM
Xiang suggested you use the master branch for brokers and the kinesis branch for everything else right? did you try that?
can you operate in that mode until you are testing? we will investigate this before publishing kinesis stuff to master
j

Jinwei Zhu

01/19/2021, 8:38 PM
Got it. Thanks
Any ideas on how to do that? should I only redeploy all the brokers using some broker images?
d

Daniel Lavoie

01/19/2021, 8:58 PM
You can use
kubectl edit pod
to change the container image tag to latest for broker.
x

Xiang Fu

01/19/2021, 9:06 PM
kubectl edit statefulset 🙂
d

Daniel Lavoie

01/19/2021, 9:07 PM
you got me 🙂
n

Neha Pawar

01/20/2021, 5:48 PM
were you able to get it running @Jinwei Zhu ?
j

Jinwei Zhu

01/20/2021, 5:54 PM
Yeah, with the master branch for brokers, I can run it
n

Neha Pawar

01/20/2021, 5:55 PM
were you able to setup the kinesis ingestion?