Hello, Looks like lot of people deployed DataHub on EKS already. I am in the process of doing the same. I am having an issue with

kafka-setup-job

pod when configuring Datahub to work with SSL AWS MSK. Is there an example of how to create

datahub-certs

secret on K8s? I found one for creating mysql and elasticsearch secrets in Datahub docs but not for SSL secrets for Kafka. If anyone could give me a sample snippet of how to create

datahub-certs

secret would help a lot. Thanks in advance!

Also experiencing this right now, help would be much appreciated!

@early-lamp-41924 Can you point these kind folks in the right direction ?

Hi folks. Here is a script we use to create the secret. See if this is applicable in your ecosystem as well!

TRUSTSTORE_PASSWORD=$(pwgen -s -1 14)
  KEYSTORE_PASSWORD=$(pwgen -s -1 14)
  cp ${JAVA_HOME}/lib/security/cacerts kafka.client.truststore.jks
  keytool -storepasswd -keystore kafka.client.truststore.jks -storepass changeit -new ${TRUSTSTORE_PASSWORD}
  keytool -genkey -keystore kafka.client.keystore.jks -validity 300 -storepass ${KEYSTORE_PASSWORD} -dname "CN=${namespace}" -alias ${namespace} -storetype pkcs12
  keytool -keystore kafka.client.keystore.jks -certreq -file client-cert-sign-request -alias ${namespace} -storepass ${KEYSTORE_PASSWORD} -keypass ${KEYSTORE_PASSWORD}
  sed -i -e "s/NEW CERTIFICATE REQUEST/CERTIFICATE REQUEST/g" client-cert-sign-request
  PRIVATE_CA_ARN=$(aws ssm get-parameters --region us-west-2 --name "/private-ca/root/arn" --with-decryption --no-cli-pager --query "Parameters[*].{Value:Value}" --output text | tr -d "\n")
  CERTIFICATE_ARN=$(aws acm-pca issue-certificate --region us-west-2 --certificate-authority-arn ${PRIVATE_CA_ARN} --csr <fileb://client-cert-sign-request> --signing-algorithm "SHA256WITHRSA" --validity Value=300,Type="DAYS" | jq -r '.CertificateArn')
  sleep 10 ## Sleep to make sure the issue certificate command has finished
  aws acm-pca get-certificate --region us-west-2 --certificate-authority-arn ${PRIVATE_CA_ARN} --certificate-arn ${CERTIFICATE_ARN} | jq -r '[.Certificate, .CertificateChain] | join("\n")' > signed-certificate-from-acm
  keytool -keystore kafka.client.keystore.jks -import -file signed-certificate-from-acm -alias ${namespace} -storepass ${KEYSTORE_PASSWORD} -keypass ${KEYSTORE_PASSWORD} -noprompt
  kubectl create secret generic ssl-config --from-file=keystore=./kafka.client.keystore.jks --from-file=truststore=./kafka.client.truststore.jks --from-literal=keystore_password=$KEYSTORE_PASSWORD --from-literal=truststore_password=$TRUSTSTORE_PASSWORD --namespace ${namespace}

Thanks for the above @early-lamp-41924 ! I am trying to use AWS MSK for Kafka and having issues with

kafka-setup-job

pod. It doesnt seem to pickup these(SASL and IAM config) values as mentioned here - https://datahubproject.io/docs/how/kafka-config#kafka

Ah so for that

Was going through this - https://github.com/datahub-project/datahub/tree/master/docker/kafka-setup and I feel I need to build a custom docker image for this

kafka-setup-job

to pickup these new env variables

Here is the script that is running https://github.com/datahub-project/datahub/blob/master/docker/kafka-setup/kafka-setup.sh Feel free to create a PR to add in the parameters you need

throwing those variables into this will pick it up for example

springKafkaConfigurationOverrides:
  security.protocol: SSL

Do we need to do this for all the components connecting to Kafka?

No. Just kafka-setup

it seems like the helm templates pick it up

For GMS / consumers, we use spring kafka library, which covers most of the kafka env variables already

Yep I tried adding those and the kafka-setup script is not picking those up but kafka-pod picks them up from Helm

😞 So what are the implications here? If you are using SSL you cannot properly create the topics?

I think we are getting mixed responses here, which is confusing the conversation

When you describe the pod -

Environment:                                                                                                                                                                                                                                           
│       KAFKA_ZOOKEEPER_CONNECT:                                 <redacted>                                                                                                                                                                                                                                                    
│       KAFKA_BOOTSTRAP_SERVER:                                  <redacted>                                                           │
│       KAFKA_PROPERTIES_KAFKASTORE_SECURITY_PROTOCOL:           SSL                                                                                                                                                                                         
│       KAFKA_PROPERTIES_KAFKASTORE_SSL_TRUSTSTORE_LOCATION:     /mnt/datahub/certs/datahub.linkedin.com.truststore.jks                                                                                                                                      
│       KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS:     software.amazon.msk.auth.iam.IAMClientCallbackHandler                                                                                                                                       
│       KAFKA_PROPERTIES_SASL_JAAS_CONFIG:                       software.amazon.msk.auth.iam.IAMLoginModule required;                                                                                                                                       
│       KAFKA_PROPERTIES_SASL_MECHANISM:                         AWS_MSK_IAM                                                                                                                                                                                 
│       KAFKA_PROPERTIES_SECURITY_PROTOCOL:                      SASL_SSL                                                                                                                                                                                    
│       KAFKA_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:                                                                                                                                                                                              
│       KAFKA_PROPERTIES_SSL_KEYSTORE_LOCATION:                  /mnt/datahub/certs/datahub.linkedin.com.keystore.jks                                                                                                                                        
│       KAFKA_PROPERTIES_SSL_KEYSTORE_TYPE:                      JKS                                                                                                                                                                                         
│       KAFKA_PROPERTIES_SSL_PROTOCOL:                           TLS                                                                                                                                                                                         
│       KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION:                /mnt/datahub/certs/datahub.linkedin.com.truststore.jks                                                                                                                                      
│       KAFKA_PROPERTIES_SSL_TRUSTSTORE_TYPE:                    JKS                                                                                                                                                                                         
│       PARTITIONS:                                              2                                                                                                                                                                                           
│       REPLICATION_FACTOR:                                      2

Yes helm tries to set KAFKA_PROPERTIES_* on the kafka-setup-job based on the values.yaml above, but it does not get picked up if it is not being read on the script

Yep was just clarifying this theory. Thanks!

Continuing this Thread, What would we do if we arent using ACM with AWS MSK but still want to use TLS/SSL with Kafka. What would be the settings/configuration there?

I am curious to know about this too, for further troubleshooting SSL issues I might face. Please let us know once you get a chance. Thanks!

main reasoning behind that is creating a certificate in ACM is $400 a month

Oh that was just an example

still want to use SSL, just not have to provide a Cert

The above script I shared uses this to authenticate https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.html

fwiw, when i just set

springKafkaConfigurationOverrides:   
  security.protocol: SSL
  kafkastore.security.protocol: SSL
  ssl.protocol: TLS
  ssl.endpoint.identification.algorithm: ""

the kafka_setup job complains that the keystore and truststore files cant be found, even though they are set to empty

Hmn. I can’t seem to find any docs on creating client auth without ACM for SSL

That is for rest

gotcha, makes sense now