Hi Everyone, I'm running Datahub v0.11 and have a working Datahub install with EKS and NodeGroups (Ec2). I'm trying to move the entire cluster to EKS Fargate (serverless). Kafka and Zookeeper need persistent storage, and I've set up and EFS file system for those two pods. The pre-requisites pods start and stay running, but Kafka crashes from time to time. When I run the datahub helms, I keep seeing this error in the logs and the kafka setup job keeps failing. Does anyone have any troubleshooting ideas?

[2024-02-21 22:46:41,386] WARN [LegacyAdminClient clientId=admin-1] Bootstrap broker prerequisites-kafka:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

Exception in thread "main" java.lang.RuntimeException: Request METADATA failed on brokers List(prerequisites-kafka:9092 (id: -1 rack: null))at kafka.admin.BrokerApiVersionsCommand$AdminClient.sendAnyNode(BrokerApiVersionsCommand.scala:156)

at kafka.admin.BrokerApiVersionsCommand$AdminClient.findAllBrokers(BrokerApiVersionsCommand.scala:179)

at kafka.admin.BrokerApiVersionsCommand$AdminClient.awaitBrokers(BrokerApiVersionsCommand.scala:171)

at kafka.admin.BrokerApiVersionsCommand$.execute(BrokerApiVersionsCommand.scala:61)

at kafka.admin.BrokerApiVersionsCommand$.main(BrokerApiVersionsCommand.scala:55)

at kafka.admin.BrokerApiVersionsCommand.main(BrokerApiVersionsCommand.scala)

it looks like kafka brokers are not running, or not accepting connections. check logs in the kafka pod

Thank you, it's working now

Hi @few-kite-46794 I 'm also facing the same issue and upon checking the

prerequisites-kafka-0

there is connection issue. How can I fix this issue?

Please provide your deployment details and the connection error you are seeing from the logs.

Hey I fixed the issue. It was due to wrong namespace pointed by kafka 🙂