Hi @User I'm working with @User and trying to use our new Pinot Kinesis support. Want to know do we have any images built on that? Because with the branch, we can not use it directly. Thanks

we haven’t built any images. You could build one from the branch: https://docs.pinot.apache.org/operators/tutorials/build-docker-images

Thanks Neha, I tried it according to the doc and got the error

[ERROR] Failed to execute goal com.mycila:license-maven-plugin:2.8:check (default) on project pinot-kinesis: Some files do not have the expected license header -> [Help 1]

[ERROR]

 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

 

[ERROR] For more information about the errors and possible solutions, please read the following articles:

[ERROR] [Help 1] <http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException>

[ERROR]

 

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR]  mvn <args> -rf :pinot-kinesis

The command '/bin/sh -c git clone ${PINOT_GIT_URL} ${PINOT_BUILD_DIR} &&   cd ${PINOT_BUILD_DIR} &&   git checkout ${PINOT_BRANCH} &&   mvn install package -DskipTests -Pbin-dist -Pbuild-shaded-jar -Dkafka.version=${KAFKA_VERSION} &&   mkdir -p ${PINOT_HOME}/configs &&   mkdir -p ${PINOT_HOME}/data &&   cp -r pinot-distribution/target/apache-pinot-*-bin/apache-pinot-*-bin/* ${PINOT_HOME}/. &&   chmod +x ${PINOT_HOME}/bin/*.sh' returned a non-zero code: 1

do you know how to deal with it?

looks like some files are missing license headers. I’ll add them and update the branch

Hi Neha, I deployed the new image and seems like the UI of Pinot doesn't running by using the new image. Do you have any ideas with that?

that is strange. that should not happen.. the kinesis branch is only a few commits behind master, and the UI code was merged long time ago

The UI is a blank page, when I switch back to the official image, the UI come back gain.

did you try clearing browser cache ?

Just worked! Thanks Neha, but one more question, why brokers all dead with this image. I redeployed few times but brokers keep dead

Why do you say they are dead? You’re seeing errors when you try to query? Sometimes the cluster can take a few minutes to be all active

Screen Shot 2021-01-14 at 3.32.37 PM.png

hmm, can you check broker logs? are servers alive?

In the broker logs, there is an error like this

2021/01/14 17:10:27.612 WARN [ClientCnxn] [Start a Pinot [BROKER]-SendThread(wb-metrics-zookeeper.wb-metrics.svc.cluster.local:2181)] Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_275]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716) ~[?:1.8.0_275]
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361) ~[pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-efbeea2bf8159eb7299292929a57c47345feb341]
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) [pinot-all-0.7.0-SNAPSHOT-jar-with-dependencies.jar:0.7.0-SNAPSHOT-efbeea2bf8159eb7299292929a57c47345feb341]

so brokers are not able to connect to Zookeeper? Are other components able to start up fine like servers and controller?

Yes, others are able to start up fine

hmm can you try to delete the broker pods and see it can fix the issue

Yes, i tried deleting them, when they start again, still shown dead in the UI

Hmm, any thing found in broker logs

So wired the logs looks good.

2021/01/15 18:28:51.181 INFO [HelixBrokerStarter] [Start a Pinot [BROKER]] Starting Pinot broker
2021/01/15 18:28:51.189 INFO [HelixBrokerStarter] [Start a Pinot [BROKER]] Connecting spectator Helix manager
2021/01/15 18:28:52.241 INFO [HelixBrokerStarter] [Start a Pinot [BROKER]] Setting up broker request handler
2021/01/15 18:28:54.144 INFO [HelixBrokerStarter] [Start a Pinot [BROKER]] Starting broker admin application on port: 8099
Jan 15, 2021 6:28:58 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [0.0.0.0:8099]
Jan 15, 2021 6:28:58 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.

hmm

No, they are not showing there

hmm, are those broker containers are running ? can you enter into the container and see the pinotBroker.log file

Hi @Neha Pawar I just debugged with Xiang and seems there are some problems in the kinesis branch. Because when switch back to the original image, it works well.

did you figure out what’s the exact problem? was there nothing in the logs?

could be

i’m able to start all components separately, locally just fine. I’ve been testing it that way using StartZK, StartController StartBroker and StartServer locally

hmm

sharded_consumer_type_support_with_kinesis

@Jinwei Zhu can you try to deploy this image:

apachepinot/pinot:sharded_consumer_type_support_with_kinesis

it worked for you @Xiang Fu ?

Hmm, seems the same as before.😔

Xiang suggested you use the master branch for brokers and the kinesis branch for everything else right? did you try that?

Got it. Thanks

You can use

kubectl edit pod

to change the container image tag to latest for broker.

kubectl edit statefulset 🙂

you got me 🙂

were you able to get it running @Jinwei Zhu ?

Yeah, with the master branch for brokers, I can run it

were you able to setup the kinesis ingestion?