Team...I am working on deploying datahub using AWS...
# all-things-deploymentn
nutritious-bird-77396
12/16/2021, 6:34 PMTeam...I am working on deploying datahub using AWS Stack - AWS OpenSearch, AWS MSK, RDS.
I am having some challenges in connecting to MSK.
MSK Cluster has TLS Encryption enabled.
Do i have to set all these variables in config?
Copy code
# KAFKA_PROPERTIES_SECURITY_PROTOCOL=SSL
# KAFKA_PROPERTIES_SSL_KEYSTORE_LOCATION=
# KAFKA_PROPERTIES_SSL_KEYSTORE_PASSWORD=
# KAFKA_PROPERTIES_SSL_KEY_PASSWORD=
# KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION=
# KAFKA_PROPERTIES_SSL_TRUSTSTORE_PASSWORD=
# KAFKA_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=teamwork 2
nutritious-bird-77396
12/16/2021, 7:14 PMOne option i see is to create a docker file to base out of datahub image and add https://github.com/aws/aws-msk-iam-auth/releases/tag/v1.1.1 to it.
I would love to hear from the community on other options.
o
orange-night-91387
12/16/2021, 11:21 PMThere's a guide for setting it up using the helm chart yml properties here (including TLS config): https://datahubproject.io/docs/deploy/aws#managed-streaming-for-apache-kafka-msk I think that's the current recommended approach. If you can figure out another approach using that library contributions to the docs are welcome š
n
nutritious-bird-77396
12/17/2021, 1:02 AMThanks Ryan.
I saw that document, digging thru the https://github.com/aws/aws-msk-iam-auth will definitely contribute once i figure out how it works.
nutritious-bird-77396
01/16/2022, 3:23 PMJust to close this thread. This issue has been solved by this PR -Ā https://github.com/linkedin/datahub/pull/3872
š 2
f
fancy-thailand-73281
06/10/2022, 5:44 PMHi Arun as per above PR the AWS_MSK_IAM is fixed.. but when I'm trying to deploy with helm charts and I'm using version "v0.8.36" we are getting errors here is the output.
could please help me and thanks
Copy code
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:433) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298) ā
ā at controllers.TrackingController.createKafkaProducer(TrackingController.java:134) ā
ā at controllers.TrackingController.<init>(TrackingController.java:52) ā
ā at controllers.TrackingController$$FastClassByGuice$$c3b1bcca.newInstance(<generated>) ā
ā at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ā
ā at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) ā
ā at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ā
ā at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ā
ā at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) ā
ā at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) ā
ā at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) ā
ā at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ā
ā at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ā
ā at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) ā
ā at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1086) ā
ā at play.api.inject.guice.GuiceInjector.instanceOf(GuiceInjectorBuilder.scala:435) ā
ā at play.api.inject.ContextClassLoaderInjector.$anonfun$instanceOf$3(Injector.scala:116) ā
ā at play.api.inject.ContextClassLoaderInjector.withContext(Injector.scala:124) ā
ā at play.api.inject.ContextClassLoaderInjector.instanceOf(Injector.scala:116) ā
ā at play.api.inject.RoutesProvider.$anonfun$get$2(BuiltinModule.scala:116) ā
ā at scala.Option.fold(Option.scala:251) ā
ā at play.api.inject.RoutesProvider.get$lzycompute(BuiltinModule.scala:116) ā
ā at play.api.inject.RoutesProvider.get(BuiltinModule.scala:111) ā
ā at play.api.inject.RoutesProvider.get(BuiltinModule.scala:105) ā
ā at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85) ā
ā at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77) ā
ā at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:59) ā
ā at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61) ā
ā at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) ā
ā ... 27 common frames omitted ā
ā Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class: software.amazon.msk.auth.iam.IAMLoginModule ā
ā at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:160) ā
ā at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ā
ā at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ā
ā at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:441) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:422) ā
ā ... 56 common frames omitted ā
ā Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: software.amazon.msk.auth.iam.IAMLoginModule ā
ā at javax.security.auth.login.LoginContext.invoke(LoginContext.java:794) Copy code
springKafkaConfigurationOverrides:
#ssl.keystore.location: /mnt/datahub/certs/datahub.linkedin.com.keystore.jks
ssl.truststore.location: /mnt/datahub/certs/kafka.client.truststore.jks
#kafkastore.ssl.truststore.location: /mnt/datahub/certs/kafka.client.truststore.jks
security.protocol: SASL_SSL
sasl.mechanism: AWS_MSK_IAM
sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
#kafkastore.security.protocol: SSL
#ssl.keystore.type: JKS
ssl.truststore.type: JKS
#ssl.protocol: TLS
#ssl.endpoint.identification.algorithm:n
nutritious-bird-77396
06/10/2022, 7:07 PMHey @fancy-thailand-73281 could you try directly using
gmssoftware.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;f
fancy-thailand-73281
06/13/2022, 4:22 PM@billions-morning-53195 š
fancy-thailand-73281
06/13/2022, 4:53 PMHi @nutritious-bird-77396 thanks for the response. I'm trying to Run locally with Docker run here are the env variables i am passing still facing issue any help and suggestion will help me a lot
Copy code
docker run -it -e MAE_CONSUMER_ENABLED='true'
-e MCE_CONSUMER_ENABLED='true'
-e SCHEMA_REGISTRY_TYPE='AWS_GLUE'
-e AWS_GLUE_SCHEMA_REGISTRY_REGION='us-east-1'
-e AWS_GLUE_SCHEMA_REGISTRY_NAME='datahub'
-e ENTITY_REGISTRY_CONFIG_PATH='/datahub/datahub-gms/resources/entity-registry.yml'
-e ANALYTICS_ENABLED='true'
-e ELASTICSEARCH_HOST='xxxxxx'
-e KAFKA_BOOTSTRAP_SERVER='xxxxxx'
-e ELASTICSEARCH_PORT='443'
-e ELASTICSEARCH_USE_SSL='true'
-e EBEAN_DATASOURCE_USERNAME='xxx'
-e EBEAN_DATASOURCE_PASSWORD='xxx'
-e EBEAN_DATASOURCE_URL='jdbc:<mysql://xxxxxx:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8&enabledTLSProtocols=TLSv1.2>'
-e EBEAN_DATASOURCE_DRIVER='com.mysql.jdbc.Driver'
-e SPRING_KAFKA_PROPERTIES_SECURITY_PROTOCOL='SASL_SSL'
-e SPRING_KAFKA_PROPERTIES_SASL_MECHANISM='AWS_MSK_IAM'
-e SPRING_KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION='/tmp/kafka/kafka.client.truststore.jks'
-e SPRING_KAFKA_PROPERTIES_SASL_JAAS_CONFIG='software.amazon.msk.auth.iam.IAMLoginModule required;'
-e SPRING_KAFKA_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS='software.amazon.msk.auth.iam.IAMClientCallbackHandler' linkedin/datahub-gms Copy code
2022/06/13 16:46:55 Waiting for: tcp:
2022/06/13 16:46:55 Waiting for: <tcp://xxxx:9098>
2022/06/13 16:46:55 Waiting for: <tcp://xxxx:9098>
2022/06/13 16:46:55 Waiting for: <https://xxxxxxx:443>
2022/06/13 16:46:55 Waiting for: http:
2022/06/13 16:46:55 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/13 16:46:55 Problem with dial: dial tcp: missing address. Sleeping 1s
2022/06/13 16:46:55 Connected to <tcp://xxxxxx:9098>
2022/06/13 16:46:55 Connected to <tcp://xxxxx:9098>
2022/06/13 16:46:55 Received 200 from <https://xxxxxxx:443>
2022/06/13 16:46:56 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/13 16:46:56 Problem with dial: dial tcp: missing address. Sleeping 1s
2022/06/13 16:46:57 Problem with dial: dial tcp: missing address. Sleeping 1s
2022/06/13 16:46:57 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/13 16:46:58 Problem with dial: dial tcp: missing address. Sleeping 1s
2022/06/13 16:46:58 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/13 16:46:59 Problem with dial: dial tcp: missing address. Sleeping 1s
2022/06/13 16:46:59 Problem with request: Get http:: http: no Host in request URL. Sleeping 1s
2022/06/13 16:47:00 Problem with dial: dial tcp: missing address. Sleeping 1sfancy-thailand-73281
06/13/2022, 4:57 PMi'm missing anything here?
n
nutritious-bird-77396
06/13/2022, 5:08 PMEnv variables looks similar to what i have did you enable debug and check for IAM?
SPRING_KAFKA_PROPERTIES_SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required awsDebugCreds=true;f
fancy-thailand-73281
06/13/2022, 5:47 PMcouldn't see other than this logs after enabling awsdebugcreds=true couldn't see other than this
Copy code
ā Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:433) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298) ā
ā at controllers.TrackingController.createKafkaProducer(TrackingController.java:134) ā
ā at controllers.TrackingController.<init>(TrackingController.java:52) ā
ā at controllers.TrackingController$$FastClassByGuice$$c3b1bcca.newInstance(<generated>) ā
ā at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ā
ā at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) ā
ā at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ā
ā at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ā
ā at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) ā
ā at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) ā
ā at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) ā
ā at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ā
ā at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ā
ā at play.api.inject.RoutesProvider.get$lzycompute(BuiltinModule.scala:116) ā
ā at play.api.inject.RoutesProvider.get(BuiltinModule.scala:111) ā
ā at play.api.inject.RoutesProvider.get(BuiltinModule.scala:105) ā
ā at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85) ā
ā at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77) ā
ā at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:59) ā
ā at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61) ā
ā at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) ā
ā ... 27 common frames omitted ā
ā Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class: software.amazon.msk.auth.iam.IAMLoginModule ā
ā at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:160) ā
ā at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ā
ā at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ā
ā at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:441) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:422) ā
ā ... 56 common frames omitted ā
ā Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: software.amazon.msk.auth.iam.IAMLoginModule ā
ā at javax.security.auth.login.LoginContext.invoke(LoginContext.java:794) ā
ā at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ā
ā at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ā
ā at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ā
ā at java.security.AccessController.doPrivileged(Native Method) ā
ā at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ā
ā at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ā
ā at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60) ā
ā at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:61) ā
ā at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:104) ā
ā at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:149)n
nutritious-bird-77396
06/13/2022, 10:06 PMThis is strange..I am able to connect to IAM using the gms built jar.
nutritious-bird-77396
06/13/2022, 10:10 PMTo help in debugging check if the msk IAM Jar is present in your container by extracting the gms lib.
https://github.com/datahub-project/datahub/blob/master/build.gradle#L52
f
fancy-thailand-73281
06/14/2022, 4:12 PMThank you for response, looks like the Jar is available on the container by executing below command in the container .. This is what you looking for?
Copy code
bash-5.1$ find / -iname '*aws-msk-iam-auth*'
find: /proc/tty/driver: Permission denied
find: /root: Permission denied
/tmp/jetty-0_0_0_0-8080-war_war-_-any-6885139113642476173/webapp/WEB-INF/lib/aws-msk-iam-auth-1.1.1.jarn
nutritious-bird-77396
06/14/2022, 4:21 PMLooking at the IAM debug logs -
2022/06/13 16:46:56 Problem with request: Get http:: http: no Host in request URL. Sleeping 1sf
fancy-thailand-73281
06/14/2022, 4:23 PMThis errors is when we tried to RUN from locally
Copy code
2022/06/13 16:46:56 Problem with request: Get http:: http: no Host in request URL. Sleeping 1sfancy-thailand-73281
06/14/2022, 4:23 PM Copy code
ā Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:433) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298) ā
ā at controllers.TrackingController.createKafkaProducer(TrackingController.java:134) ā
ā at controllers.TrackingController.<init>(TrackingController.java:52) ā
ā at controllers.TrackingController$$FastClassByGuice$$c3b1bcca.newInstance(<generated>) ā
ā at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ā
ā at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114) ā
ā at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ā
ā at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ā
ā at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:42) ā
ā at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:65) ā
ā at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:113) ā
ā at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91) ā
ā at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306) ā
ā at play.api.inject.RoutesProvider.get$lzycompute(BuiltinModule.scala:116) ā
ā at play.api.inject.RoutesProvider.get(BuiltinModule.scala:111) ā
ā at play.api.inject.RoutesProvider.get(BuiltinModule.scala:105) ā
ā at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:85) ā
ā at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:77) ā
ā at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:59) ā
ā at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61) ā
ā at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1050) ā
ā ... 27 common frames omitted ā
ā Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: unable to find LoginModule class: software.amazon.msk.auth.iam.IAMLoginModule ā
ā at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:160) ā
ā at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:146) ā
ā at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67) ā
ā at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:99) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:441) ā
ā at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:422) ā
ā ... 56 common frames omitted ā
ā Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: software.amazon.msk.auth.iam.IAMLoginModule ā
ā at javax.security.auth.login.LoginContext.invoke(LoginContext.java:794) ā
ā at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195) ā
ā at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682) ā
ā at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680) ā
ā at java.security.AccessController.doPrivileged(Native Method) ā
ā at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680) ā
ā at javax.security.auth.login.LoginContext.login(LoginContext.java:587) ā
ā at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:60) ā
ā at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:61) ā
ā at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:104) ā
ā at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:149)fancy-thailand-73281
06/14/2022, 4:25 PMWhen we deploy with Helm in GMS pod we run the below command
Copy code
bash-5.1$ find / -iname '*aws-msk-iam-auth*'
find: /proc/tty/driver: Permission denied
find: /root: Permission denied
/tmp/jetty-0_0_0_0-8080-war_war-_-any-6885139113642476173/webapp/WEB-INF/lib/aws-msk-iam-auth-1.1.1.jarfancy-thailand-73281
06/14/2022, 4:25 PMWe see the IAM lib with find command in the GMS POD but we still see its not available in logs
b
billions-morning-53195
06/14/2022, 5:08 PM@early-lamp-41924 this is the thread
m
melodic-market-88762
06/22/2022, 2:46 PM@calm-dinner-63735 take a look at this.
c
calm-dinner-63735
06/22/2022, 2:53 PM@melodic-market-88762 but i am using Plaintext , not any IAM_AUTH
8 Views