Is there any doc on how to setup kafka on confluen...
# all-things-deployments
square-activity-64562
07/08/2021, 10:20 AMIs there any doc on how to setup kafka on confluent cloud (https://confluent.cloud/) with datahub? I read https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub and it just mentions the URLs under
global.kafka.*square-activity-64562
07/08/2021, 10:24 AMfound something at https://github.com/linkedin/datahub/issues/2504. Hopefully this works
square-activity-64562
07/08/2021, 10:38 AMIt is not clear what to do with zookeeper with confluent cloud. Don't think datahub needs direct zookeeper access. I am going to try to run it using the values edited from https://github.com/linkedin/datahub/blob/master/datahub-kubernetes/datahub/values.yaml
square-activity-64562
07/08/2021, 11:21 AMGetting this error in
datahub-kafka-setup-job Copy code
[kafka-admin-client-thread | adminclient-1] ERROR org.apache.kafka.common.utils.KafkaThread - Uncaught exception in thread 'kafka-admin-client-thread | adminclient-1':
java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30) at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:112) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385) at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572) at org.apache.kafka.common.network.Selector.poll(Selector.java:483) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:540) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1196) at java.lang.Thread.run(Thread.java:748)square-activity-64562
07/09/2021, 5:56 AMAny help on this would be great. Still seeing this with chart version
0.2.6values.yml Copy code
kafka:
bootstrap:
server: "KEY_ID:KEY_SECRET@CLUSTER_IDENTIFIER.asia-southeast1.gcp.confluent.cloud:9092"
zookeeper:
server: "prerequisites-zookeeper:2181"
schemaregistry:
url: "KEY_ID:KEY_SECRET@CLUSTER_IDENTIFIER.australia-southeast1.gcp.confluent.cloud"square-activity-64562
07/09/2021, 6:00 AM@mammoth-bear-12532 @early-lamp-41924 any suggestions for this?
square-activity-64562
07/12/2021, 7:32 AMI see that it is simply creating the 4 topics. I can do that manually in confluent cloud and disable this job
m
mammoth-bear-12532
07/14/2021, 6:26 AM
Thanks for letting us know @square-activity-64562 , there is probably a way to do this using the admin client that uses the broker connection instead of Zk. Will have to research that.
s
square-activity-64562
07/14/2021, 6:48 AMSome other improvements related to kafka auth to confluent cloud
https://datahubspace.slack.com/archives/CV2UVAPPG/p1626245271249900?thread_ts=1626093519.225800&cid=CV2UVAPPG
square-activity-64562
07/14/2021, 6:52 AMdoing it without zookeeper https://stackoverflow.com/a/45122955/2235567
s
straight-salesclerk-65619
07/27/2021, 9:09 PMHi @square-activity-64562 - Trying the same. Did you get it to work on Confluent Cloud without ZK?
s
square-activity-64562
07/28/2021, 4:37 AMyes was able to do that. Had to disable kafka setup job @straight-salesclerk-65619 and do some setting in helm config. @big-carpet-38439 is going to write a doc for that soon which should help you out
s
straight-salesclerk-65619
07/28/2021, 4:11 PMThanks @square-activity-64562!
b
big-carpet-38439
07/28/2021, 4:30 PMYes! Hoping to get that out shortly
big-carpet-38439
07/28/2021, 4:30 PMI'll send you what I have so far @straight-salesclerk-65619
s
straight-salesclerk-65619
07/28/2021, 5:37 PMMuch appreciated @big-carpet-38439! Thanks!
2 Views