https://datahubproject.io logo
Join Slack
Powered by
Hi! After playing with Datahub on my dev machine ...
# all-things-deployment
s
Hi! After playing with Datahub on my dev machine via docker-compose, I’m trying to set it up on GKE so my teammates canm also play with it and convince them to adopt it as our Data Catalog. However, our Kafka is on Aiven, which do not expose/require a zookeeper. It seems this doesn’t suits too well with 
datahub-kafka-setup-job
since I can see its yaml has this conf:*
Copy code
spec:                                                                                                                                                                                                                                      
│   containers:                                                                                                                                                                                                                              
│   - env:                                                                                                                                                                                                                                   
│     - name: KAFKA_ZOOKEEPER_CONNECT                                                                                                                                                                                                        
│       value: prerequisites-zookeeper:2181
which doesn’t exists obviously. It seems this is causing the job to time out. Any ideas what can be done about it? thanks
🔍 1
📖 1
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ button ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? ✅ button Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
s
mmm seems the issue is actually
Copy code
Unable to attach or mount volumes: unmounted volumes=[datahub-certs-dir], unattached volumes=[kube-api-access-2bt9w datahub-certs-dir]: timed out waiting for the condition
a
@dazzling-yak-93039 might be able to help out here vis-a-vis hooking kafka/aiven in
d
Sounds like it can't come up because it doesn't have the certificates it needs because the volumes are not mounted?
s
Hi! Sorry for the late reply. I created the secret on GKE:
with the following config:
Copy code
credentialsAndCertsSecrets:
      name: datahub-certs
      path: /mnt/datahub/certs
      secureEnv:
        <http://basic.auth.user.info|basic.auth.user.info>: sasl-auth
        sasl.username: sasl-username
        sasl.password: sasl-password
  springKafkaConfigurationOverrides:
      sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="myuser" password="mypass";
      sasl.mechanism: PLAIN
      security.protocol: SASL_PLAINTEXT
      ssl.endpoint.identification.algorith: https
      client.dns.lookup: use_all_dns_ips
      basic.auth.credentials.source: USER_INFO
However the kafkaSetupJob starts with 
sasl.mechanism
= 
GSSAP
which iiuc is for kerberos, my 
sasl.jaas.config
is for PLAIN not for kerberos so of course the job fails
Copy code
[main] INFO org.apache.kafka.common.security.authenticator.AbstractLogin - Successfully logged in.
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'sasl.jaas.config' was supplied but isn't a known config.
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'sasl.kerberos.service.name' was supplied but isn't a known config.
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 6.1.4-ccs
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: c9124241a6ff43bc
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1683294446800
[kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.common.network.SaslChannelBuilder - [AdminClient clientId=adminclient-1] Failed to create channel due to
org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.firstPrincipal(SaslClientAuthenticator.java:622)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.<init>(SaslClientAuthenticator.java:200)
        at org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:274)
        at org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:216)
        at org.apache.kafka.common.network.KafkaChannel.<init>(KafkaChannel.java:142)
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:224)
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:338)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:329)
looks like datahub (or most likely spring-kafka` doesn’t like my 
sasl.mechanism
property
envars are also ignored apparently
Copy code
helm install datahub -f datahub/aiven-neo4j.yaml --set KAFKA_PROPERTIES_SASL_MECHANISM=$SPRING_KAFKA_PROPERTIES_SASL_MECHANISM --set KAFKA_PROPERTIES_SASL_KERBEROS_SERVICE_NAME=$SPRING_KAFKA_PROPERTIES_SASL_KERBEROS_SERVICE_NAME datahub
Screenshot 2023-05-08 at 13.45.52.png
a
Hey @square-football-37770, is this issue still effecting you?
s
I created the topics manually. 
datahub-actions
fails while `gms`and 
frontend
seems to start alright and I even see some consumers connected in AIven
image.png
but then kafka ingestion seems to fail
which fails with:
Copy code
Exception: Failed to instantiate Actions Pipeline using config {'name': 'ingestion_executor', 'source': {'type': 'kafka', 'config': {'connection': {'bootstrap': '<http://kafka-01-my-instance-aiven.aivencloud.com:29658|kafka-01-my-instance-aiven.aivencloud.com:29658>', 'schema_registry_url': '<https://myuser:mypass@kafka-01-my-instance-aiven.aivencloud.com:29650>', 'consumer_config': {'security.protocol': 'SASL_SSL', 'sasl.mechanism': 'SCRAM-SHA-512', 'sasl.username': 'myuser', 'sasl.password': 'mypass', 'sasl.jaas.config': 'org.apache.kafka.common.security.scram.ScramLoginModule required username=myuser password=mypass;', 'ssl.truststore.location': '/mnt/datahub/certs/truststore.jks', 'kafkastore.ssl.truststore.location': '/mnt/datahub/certs/truststore.jks'}, 'schema_registry_config': {'<http://basic.auth.user.info|basic.auth.user.info>': 'myuser:mypass'}}, 'topic_routes': {'mcl': 'MetadataChangeLog_Versioned_v1', 'pe': 'PlatformEvent_v1'}}}, 'filter': {'event_type': 'MetadataChangeLogEvent_v1', 'event': {'entityType': 'dataHubExecutionRequest', 'changeType': 'UPSERT', 'aspectName': ['dataHubExecutionRequestInput', 'dataHubExecutionRequestSignal'], 'aspect': {'value': {'executorId': 'default'}}}}, 'action': {'type': 'executor', 'config': {'executor_id': 'default'}}, 'datahub': {'server': '<http://datahub-datahub-gms:8080>', 'extra_headers': {'Authorization': 'Basic __datahub_system:JohnSnowKnowsNothing'}}}
mmm that last 
Authorization
header is probably to blame since i haven’t setup any auth for 
gms
🤔
nope, even changing that to
Copy code
{'server': '<http://datahub-datahub-gms:8080>', 'extra_headers': {'Authorization': 'Bearer eyJhbGciOiJIUzI1NiJ9.eyJhY3RvclR5cGUiOiJVU0VSIiwiYWN0b3JJZCI6ImRhdGFodWIiLCJ0eXBlIjoiUEVSU09OQUwiLCJ2ZXJzaW9uIjoiMiIsImp0aSI6IjlmZTk1NTkxLWI5YTgtNDZkYi04YTI4LTc2NmY3ZTBhMzAyMyIsInN1YiI6ImRhdGFodWIiLCJleHAiOjE2ODY5MDg1OTYsImlzcyI6ImRhdGFodWItbWV0YWRhdGEtc2VydmljZSJ9.TNN0_Vxzu7gnYAcGmECWBX4SCrYmtsLHH3Dliqkck-w'}}
it still fails
same error
Open in Slack
PreviousNext
Powered by