https://datahubproject.io logo
Join Slack
Powered by
I’m trying to deploy DataHub in AWS using MSK. Is ...
# troubleshoot
s
I’m trying to deploy DataHub in AWS using MSK. Is it possible to configure DataHub to work with SASL_SCRAM authentication for MSK. Could you provide an example for 
values.yaml
? Thanks in advance!
e
Hey @nutritious-bird-77396 Remember you were able to set up SASL for MSK. Do you mind sharing any details on how you managed to do that?
n
@salmon-area-51650 Added all the variables that we need to pass in the Service to make SASL work for MSK - https://github.com/linkedin/datahub/blob/master/docs/how/kafka-config.md#how-to-configure-kafka Kindly take a look and let me know if you have any questions.
e
Thanks a lot @nutritious-bird-77396!!!
teamwork 1
s
Thanks @nutritious-bird-77396 and @early-lamp-41924! I assume that the configuration properties should be added to 
kafka
section in values.yaml, right?
e
Check out how https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L171 credentialsAndCertsSecrets and springKafkaConfigurationOverrides is used!
they get converted into env variables here https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-gms/templates/deployment.yaml#L162 so you can reverse engineer what you need to set in values.yaml!
s
Thanks @early-lamp-41924!! I’ll try it 🙂
I’m getting an error from 
datahub-kafka-setup
job
Copy code
Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
This is my configuration in 
values.yaml
Copy code
datahub-gms:
  enabled: true
  image:
    repository: linkedin/datahub-gms
    tag: "v0.8.26"

datahub-frontend:
  enabled: true
  image:
    repository: linkedin/datahub-frontend-react
    tag: "v0.8.26"
  # Set up ingress to expose react front-end
  ingress:
    enabled: false

datahub-mae-consumer:
  image:
    repository: linkedin/datahub-mae-consumer
    tag: "v0.8.26"

datahub-mce-consumer:
  image:
    repository: linkedin/datahub-mce-consumer
    tag: "v0.8.26"
kafkaSetupJob:
  enabled: true
  image:
    repository: linkedin/datahub-kafka-setup
    tag: "v0.8.26"

global:
  graph_service_impl: elasticsearch
  datahub_analytics_enabled: true
  datahub_standalone_consumers_enabled: false

  elasticsearch:
    host: "<http://XXXXXX.eu-west-2.es.amazonaws.com|XXXXXX.eu-west-2.es.amazonaws.com>"
    port: "443"
    useSSL: "true"

  kafka:
    bootstrap:
      server: "<http://XXXXXX.kafka.eu-west-2.amazonaws.com:9096,YYYYY.kafka.eu-west-2.amazonaws.com:9096,ZZZZZ.kafka.eu-west-2.amazonaws.com:9096|XXXXXX.kafka.eu-west-2.amazonaws.com:9096,YYYYY.kafka.eu-west-2.amazonaws.com:9096,ZZZZZ.kafka.eu-west-2.amazonaws.com:9096>"
    zookeeper:
      server: "<http://AAAAAA.kafka.eu-west-2.amazonaws.com:2181,BBBBB.kafka.eu-west-2.amazonaws.com:2181,CCCC.kafka.eu-west-2.amazonaws.com:2181|AAAAAA.kafka.eu-west-2.amazonaws.com:2181,BBBBB.kafka.eu-west-2.amazonaws.com:2181,CCCC.kafka.eu-west-2.amazonaws.com:2181>"
    partitions: 3
    replicationFactor: 3
    schemaregistry:
      url: "<http://SSSSS.kafka.svc.cluster.local>"

  sql:
    datasource:
      host: "<http://XXXXXX.eu-west-2.rds.amazonaws.com:3306|XXXXXX.eu-west-2.rds.amazonaws.com:3306>"
      hostForMysqlClient: "<http://XXXXXXX.eu-west-2.rds.amazonaws.com|XXXXXXX.eu-west-2.rds.amazonaws.com>"
      port: "3306"
      url: "jdbc:<mysql://XXXXXXX.eu-west-2.rds.amazonaws.com:3306/myuser?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>"
      driver: "com.mysql.cj.jdbc.Driver"
      username: "datahub_user"
      password:
        secretRef: mysql-credentials
        secretKey: mysql_user-password

  datahub:
    gms:
      port: "8080"
      nodePort: "30001"
    mae_consumer:
      port: "9091"
      nodePort: "30002"
    appVersion: "1.0"

    encryptionKey:
      secretRef: "datahub-encryption-secrets"
      secretKey: "encryption_key_secret"
      # Set to false if you'd like to provide your own secret.
      provisionSecret: true

    managed_ingestion:
      enabled: true
      defaultCliVersion: "0.8.26.6"

    metadata_service_authentication:
      enabled: false
      systemClientId: "__datahub_system"
      systemClientSecret:
        secretRef: "datahub-auth-secrets"
        secretKey: "token_service_signing_key"
      tokenService:
        signingKey:
          secretRef: "datahub-auth-secrets"
          secretKey: "token_service_signing_key"
      # Set to false if you'd like to provide your own auth secrets
      provisionSecrets: true

  springKafkaConfigurationOverrides:
    security.protocol: "SASL_SSL"
    sasl.mechanism: "SCRAM-SHA-512"
    sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="XXXXXXXX" password="YYYYYYYYY";
cc @early-lamp-41924 @nutritious-bird-77396 Thanks in advance!
I’ve created the topics manually, but now I’m receiving issues from 
acryl-datahub-actions
pod
Copy code
KafkaException: KafkaError{code=_INVALID_ARG,val=-186,str="Failed to create consumer: No provider for SASL mechanism GSSAPI: recompile librdkafka with libsasl2 or openssl support. Current build options: PLAIN SASL_SCRAM OAUTHBEARER"}
I think this pod is executing a Python code and that job is not picking up the authentication configuration
plus1 1
@early-lamp-41924 @nutritious-bird-77396 do you have any clue why 
acryl-datahub-actions
pod is not picking up the configuration parameters from spring properties? Also… what is 
acryl-datahub-actions
for? Thanks in advance!
n
acryl-datahub-actions
is for scheduling your ingestion jobs that is set from the UI. Its part of the actions framework. It was released part of v0.8.26 version Also as i understand from @early-lamp-41924 its a python project so it will not pickup the spring configuration params instead you should try setting the config params without 
SPRING
prefix. As actions code is not open sourced yet only the image has been published I have not gotten a chance to look into the details yet.
e
Exactly as @nutritious-bird-77396 mentioned. Spring is used only by java based applications. acryl-datahub-actions is used to support UI based ingestion right now. To enable it, you need this pod running. Just simply remove SPRING and keep everything KAFKA_PROPERTIES_* and it should work.
s
But according to this code, these 
KAFKA_PROPERTIES_
properties should be extracted from 
springKafkaConfigurationOverrides
section right?
plus1 1
r
Just to be clear, is the current workaround to define the 
KAFKA_PROPERTIES_*
env variables as 
extraEnvs
for this container in the helm chart 
values.yaml
?
^ To follow up on the above, the 
KAFKA_PROPERTIES_*
env variables are indeed set on the container, but it seems they are not picked up by the app somehow.
plus1 2
b
^ Just curious if anyone was able to debug the above - I’m hitting the same issue and I’d love any tips folks might have!
t
@salmon-area-51650 Were you able to get Datahub to work with AWS MSK? I'm trying to do the same thing and am hitting a few issues. I don't find to documentation all that clear so was wondering if picking your brain would be possible.
s
Hi @tall-fall-45442. This is my configuration for MSK:
Copy code
kafka:
    bootstrap:
      server: "broker1:9096,broker2:9096,broker3:9096"
    zookeeper:
      server: "zookeeper1:2181,zookeeper2:2181,zookeeper3:2181"
    partitions: 3
    replicationFactor: 3
    schemaregistry:
      url: "<http://schema.registry:8081>"

credentialsAndCertsSecrets:
  name: sasl-jass-config
  secureEnv:
    sasl.jaas.config: sasl_jaas_config
springKafkaConfigurationOverrides:
  security.protocol: "SASL_SSL"
  sasl.mechanism: "SCRAM-SHA-512"
There is a secret called 
sasl_jaas_config
which contains the following:
Copy code
sasl_jaas_config: org.apache.kafka.common.security.scram.ScramLoginModule required username='kafka_user' password='kafka_user_pass';
t
@salmon-area-51650 Thank. However I am now getting an error about not being able to find a KafkaClient.
Copy code
[main] ERROR io.confluent.admin.utils.cli.KafkaReadyCommand - Error while running kafka-ready.
org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:535)
        at org.apache.kafka.clients.admin.Admin.create(Admin.java:75)
        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
        at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:138)
        at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
        at org.apache.kafka.common.security.JaasContext.defaultContext(JaasContext.java:131)
        at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:96)
        at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:82)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:134)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)
        ... 4 more
s
Seems like you have an issue in your 
sasl_jaas_config
. Can you check that it’s correct? Are you sure that you have 
SASL/SCRAM
enabled in your MSK?
10 Views
Open in Slack
PreviousNext
Powered by