Hi everyone, I am trying to deploy the datahub on our custom k8s cluster (not EKS) on aws. I have provisioned managed pre-requisites services (MSK, OpenSearch and RDS). Schema-registry will still be the pod which will use MSK. Did anyone already had such scenario? I following this guide but facing a lot of issues in the deployment https://datahubproject.io/docs/deploy/aws

Have you been able to run any other application on this custom kubernetes cluster?

yes a lot of applications are being running on this k8s cluster. For example apache airflow, kafka connect (which is using MSK) etc

What kind of errors are you facing?

I deployed schema-registry (using MSK) in the prerequisites, it is unable to connect to MSK

Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list.

Hello Hamid, How many nodes does your MSK cluster have? This looks like a kafka specific issue, not datahub specific.

Do you have any SSL set up for the MSK cluster?