Hi everyone, I've tried setting up datahub via kubernetes the last two days. For kafka I use a Strimzi cluster I set up, so I don't use the built in bitnami kafka from the prerequisites. This Strimzi cluster is using mutual TLS so I need to pass the certificates to every client of Kafka. I've managed so far to set up everything and get most of the datahub components up and running on my kubernetes cluster. However the

datahub-acryl-datahub-actions

components is constantly logging the following error

%3|1646144911.225|FAIL|rdkafka#consumer-1| [thrd:<ssl://testing-strimzi-cluster-kafka-bootstrap:9092/bootstrap>]: <ssl://testing-strimzi-cluster-kafka-bootstrap:9092/bootstrap>: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 5ms in state SSL_HANDSHAKE, 31 identical error(s) suppressed)

I currently pass the following environment variables via the deployment yaml: • KAFKA_PROPERTIES_SSL_KEY_PASSWORD • KAFKA_PROPERTIES_KAFKASTORE_SSL_TRUSTSTORE_PASSWORD • KAFKA_PROPERTIES_SSL_TRUSTSTORE_PASSWORD • KAFKA_PROPERTIES_SSL_KEYSTORE_PASSWORD • KAFKA_PROPERTIES_KAFKASTORE_SSL_KEYSTORE_PASSWORD • KAFKA_PROPERTIES_SSL_TRUSTSTORE_TYPE • KAFKA_PROPERTIES_SSL_KEYSTORE_LOCATION • KAFKA_PROPERTIES_SSL_TRUSTSTORE_LOCATION • KAFKA_PROPERTIES_KAFKASTORE_SSL_TRUSTSTORE.LOCATION • KAFKA_PROPERTIES_SECURITY_PROTOCOL • KAFKA_PROPERTIES_KAFKASTORE_SECURITY_PROTOCOL • KAFKA_PROPERTIES_SSL_PROTOCOL • KAFKA_PROPERTIES_SSL_ENDPOINT_IDENTIFICATION.ALGORITHM • KAFKA_PROPERTIES_SSL_CA_LOCATION (= truststore location)

Hi - there's good news & bad news. The bad news: The actions container is a Python process, so these trust-store / key-store configurations unfortunately will not work with it. Currently, this container does not expose configuration for a custom client-side certificate to be used when connecting to Kafka. This would require new development to enable. The good news: The actions container is not mission critical in the operation of DataHub. It is used primarily to enable UI-based ingestion. However, you can still ingest from the DataHub CLI and continue to use DataHub's many other features. I'd recommend you open an issue to track this so we can prioritize this work based on wider demand from the community. Thanks!

Ah alright, good to know that it's not an important component, will open a ticket for it. Thanks a lot!

@miniature-account-72792 Please ignore the red herrings on the upgrade pod for now - Since you have a fresh install, there should no upgrade you'll need to run. We need to improve how we determine whether the upgrade job should run or not.. cc. @early-lamp-41924

Currently not, only the communication with Kafka is done via SSL

Okay then I don't see any issues here. Are you able to query http://datahub-gms:8080/config? Have you made any modifications to the default gms port of 8080?