Hi Guys, Need some Help for Datahub Deployment. We are trying to connect to an external Kafka Cluster (Authentification with mTLS). Kafka Setup Container fails on startup.

Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /mnt/datahub/certs/keystore.jks of type PKCS12
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:349)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:299)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161)
at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:138)
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:71)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:508)
... 4 more
Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=111, too big.
at sun.security.util.DerInputStream.getLength(DerInputStream.java:601)
at sun.security.util.DerValue.init(DerValue.java:384)
at sun.security.util.DerValue.<init>(DerValue.java:325)
at sun.security.util.DerValue.<init>(DerValue.java:338)
at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:1958)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:374)

Our truststore and keystore commands look like this:

keytool -noprompt -keystore truststore.jks -storetype pkcs12 -alias kafka-ca01-q -trustcacerts -import -file <ca> -deststorepass <password>
openssl pkcs12 -export -in <cert> -inkey <key> -out client.p12 -name localhost -passout pass:<password>
keytool -importkeystore -srckeystore client.p12 -srcstoretype pkcs12 -srcstorepass <password> -destkeystore ./keystore.jks -deststoretype pkcs12 -deststorepass <password> -destkeypass <password>
keytool -noprompt -keystore keystore.jks -storetype pkcs12 -alias kafka-ca01-q -import -file <ca> -storepass <password>

Hey, could you share the the part from your yaml, where you specify the crednetials ?

Hey Peter, the password is defined as k8s secret. The generated truststore and keystore is also defined as a kubectl secret. Kubectl-Secret:

kubectl create secret generic datahub-secrets --from-file=keystore.jks,truststore.jks --namespace=datahub

yaml-config:

credentialsAndCertsSecrets:
    name: datahub-secrets
    path: /mnt/datahub/certs
    secureEnv:
      ssl.key.password: password
      ssl.keystore.password: password
      ssl.truststore.password: password
      kafkastore.ssl.truststore.password: password

  springKafkaConfigurationOverrides:
    ssl.keystore.location: /mnt/datahub/certs/keystore.jks
    ssl.truststore.location: /mnt/datahub/certs/truststore.jks
    kafkastore.ssl.truststore.location: /mnt/datahub/certs/truststore.jks
    security.protocol: SSL
    kafkastore.security.protocol: SSL
    ssl.keystore.type: JKS
    ssl.truststore.type: JKS
    ssl.protocol: TLS
    ssl.endpoint.identification.algorithm:

when the pod is running, can you see the secret is mounted to it correctly?

The secrets are mounted correctly.

bash-5.1$ ls /mnt/datahub/certs/
keystore.jks  password  truststore.jks
bash-5.1$

will try to reproduce this!

Hi Peter, did you had a chance to try this? Thanks! :)

Hi @bulky-electrician-72362, we got a little progress in the meanwhile on our side. One issue that we discovered was an incorrect escaping of the Kubernetes Secret containing the Java Keystores. After many hours of fiddling with different types of Keystores we came to a setup using pkcs12 keystores specifically created with Java 8 as newer Java versions use a hash version that is not present in the JRE the datahub containers currently use resulting in the following error:

Caused by: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available

With this setup we were able to successfully pass the Kafka Prep Job. However the datahub-actions container still causes us headaches, I described the problem here: https://github.com/datahub-project/datahub/issues/5786#issuecomment-1282406012

Hey, sorry I was on holiday and totally missed this. Does this mean the keystore works for you now?

@bulky-electrician-72362 no problem, hope you had a nice one 😉 thanks for you reply. Yes the keystores are working for us in the meanwhile and as described above and in the linked issue the upgrade to Java 11 with v0.9.0 makes the handling much easier as know all modern keystore types and hashing algorithms are supported (which was not the case with Java 8 ). In my last comment on the GitHub issue linked above I raised some questions that are currently bothering us with regards to keystores, certificate files and configs that need to be maintained in the current setup. Can you say anything about those or point us to the correct person to discusses these? There are some areas that need to be optimized when it comes to TLS enabled environments in my opinion.

Glad it's working for you now. Will tag the relevant people on the issue. Thanks for raising this.