https://datahubproject.io logo
Join Slack
Powered by
Hi guys, when trying to deploy prerequisites with ...
# all-things-deployment
b
Hi guys, when trying to deploy prerequisites with helm cp-schema-registry pod is stuck on CrashLoopBackOff. Looking at the pod logs I see this error:
Copy code
ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1661848715277, tries=1, nextAllowedTryMs=1661848715378) timed out at 1661848715278 after 1 attempt(s)
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
	at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:149)
	at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1661848715277, tries=1, nextAllowedTryMs=1661848715378) timed out at 1661848715278 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[main] INFO io.confluent.admin.utils.ClusterStatus - Expected 1 brokers but found only 0. Trying to query Kafka for metadata again ...
[main] ERROR io.confluent.admin.utils.ClusterStatus - Expected 1 brokers but found only 0. Brokers found [].
Can anyone assist please?
b
could you confirm if kafka pod is running healthy . Error say unknown host excetion -check if kafka bootstrap url is correct in values.yaml .
also share the output of 
kubectl get svc -n <namespace>
b
mmm I get it.. need to change to the release name
b
check if it is matching service name for kafka
b
it is
Copy code
prerequisites-kafka                                       ClusterIP   10.136.9.141    <none>        9092/TCP                     62m
prerequisites-kafka-headless                              ClusterIP   None            <none>        9092/TCP,9093/TCP            62m
b
you need to correct bootstrap url in datahub values.yaml file
b
changed to
Copy code
bootstrapServers: "prerequisites-kafka-headless:9092"
but pod still stuck
.
b
what error are you getting now
b
same error
might be persistence volume size issue?
b
shouldn't be ...still it says 0 broker found
b
kafka is not running healthy
b
that explains it .. i asked about it health earlier
b
sorry I missed that. I’ve shared the kafka pod log in the beginning
b
could you hare more detailed logs from the pod
b
Copy code
[2022-08-30 09:48:53,253] WARN [Controller id=0, targetBrokerId=0] Error connecting to node prerequisites-kafka-0.prerequisites-kafka-headless.prod-it-data.svc.cluster.local:9093 (id: 0 rack: null) (org.apache.kafka.clients.NetworkClient)
java.net.UnknownHostException: prerequisites-kafka-0.prerequisites-kafka-headless.prod-it-data.svc.cluster.local
	at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
	at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1509)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1368)
	at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1302)
	at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27)
	at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511)
	at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:468)
	at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173)
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:988)
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:301)
	at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:64)
	at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:291)
	at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:245)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2022-08-30 09:48:53,253] WARN [RequestSendThread controllerId=0] Controller 0's connection to broker prerequisites-kafka-0.prerequisites-kafka-headless.prod-it-data.svc.cluster.local:9093 (id: 0 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to prerequisites-kafka-0.prerequisites-kafka-headless.prod-it-data.svc.cluster.local:9093 (id: 0 rack: null) failed.
	at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)
	at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:291)
	at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:245)
	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
[2022-08-30 09:48:53,253] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from node 0 (org.apache.kafka.clients.NetworkClient)
b
if not , i would suggest to uninstall helm and reinstall it using correct release name
b
I set clusterDomain to be ours instead of the default “local” in the values.yaml and it worked. thanks 🙏
2 Views
Open in Slack
PreviousNext
Powered by