Hi Team I am seeing the following errors in our systemUpdate DataHub #all-things-deployment

Hi Team - I am seeing the following errors in our ...

ripe-agency-8696

09/28/2023, 8:30 PM

Hi Team - I am seeing the following errors in our systemUpdate job

Copy code

2023-09-28 19:19:48,699 [main] ERROR c.l.m.d.producer.KafkaHealthChecker:34 - Failed to emit History Event for entity Event Version: v0.10.5-1
org.apache.kafka.common.errors.TimeoutException: Topic DataHubUpgradeHistory_v1 not present in metadata after 60000 ms.

Here is our ssl configuration. Please let me know if this needs to be added to the datahubSystemUpdate section or not?

Copy code

credentialsAndCertsSecrets:
            name: datahub-certs
            path: /ssl
            secureEnv:
              ssl.truststore.password: password
              kafkastore.ssl.truststore.password: password
    
          springKafkaConfigurationOverrides:
            ssl.truststore.location: /ssl/truststore.jks
            kafkastore.ssl.truststore.location: /ssl/truststore.jks
            sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="datahub_staging" password="${jaasconfig}";
            security.protocol: SASL_SSL
            kafkastore.security.protocol: SASL_SSL
            ssl.protocol: TLS
            ssl.truststore.type: JKS
            sasl.mechanism: SCRAM-SHA-256
            ssl.endpoint.identification.algorithm:

ripe-agency-8696

09/28/2023, 8:32 PM

when I look into the systemUpdate job log, I see the following are completely empty

Copy code

sasl.jaas.config = null
        ssl.truststore.location = null
        ssl.truststore.password = null

Can someone help in providing pointers on how to pass this onto the systemUpdate job?

ripe-agency-8696

09/28/2023, 8:38 PM

@gentle-night-56466 @green-football-43791 any suggestions?

ripe-agency-8696

09/28/2023, 8:56 PM

I confirmed by logging into our pod that the truststore file is present . but not sure why the configs are all in the log.

ripe-agency-8696

09/28/2023, 8:57 PM

FYI

Copy code

sasl.jaas.config: org.apache.kafka.common.security.scram.ScramLoginModule required username="datahub_staging" password="${jaasconfig}";

In this line

jaasconfig

is injected an env variable. into the pod

shy-dog-84302

09/29/2023, 2:45 AM

Hi @ripe-agency-8696 are you sure that

DataHubUpgradeHistory_v1

topic does not exist on your Kafka cluster? How do you install your DataHub today? Helm charts or docker setup? Have you checked kafka-setup-job logs?

DataHubUpgradeHistory_v1

should have been created by this setup job.

thanks bear 1

ripe-agency-8696

09/29/2023, 2:46 AM

hi @shy-dog-84302 - we have a managed kafka cluster and because of that, we explicitly created the kafka topics and disabled the kafka setup job. the rest of the datahub installation is all done via helm charts.

ripe-agency-8696

09/29/2023, 2:46 AM

i think the issue might be something to do with the ssl configuration.

shy-dog-84302

09/29/2023, 3:06 AM

Copy code

org.apache.kafka.common.errors.TimeoutException: Topic DataHubUpgradeHistory_v1 not present in metadata after 60000 ms.

This could mean that the

KafkaHealthChecker

is not able to contact the broker if the topic exists. How does the describe command output looks on the systemUpdateJob?

kubectl describe job datahub-system-update-job

shy-dog-84302

09/29/2023, 3:11 AM

I assume you have seen this page: https://datahubproject.io/docs/0.10.5/how/kafka-config

ripe-agency-8696

10/01/2023, 3:21 PM

Super helpful Upendra. Thanks for your help

ripe-agency-8696

10/01/2023, 3:21 PM

I will go through the docs

3 Views

Open in Slack

Previous Next