Hi team I am editing the Kafka topics inside the kafka setup DataHub #troubleshoot

Hi team, I am editing the Kafka topics inside the ...

colossal-smartphone-90274

01/19/2023, 4:40 PM

Hi team, I am editing the Kafka topics inside the kafka-setup and schema-registry but the bottom 4 topics are not being correctly set (using the default values). I am following the naming procedure that appears in the datahub-gms section of https://datahubproject.io/docs/how/kafka-config/#metadata-service-datahub-gms Note that the schema-registry extra-envs are identical to these ones:

Copy code

kafkaSetupJob:
  enabled: true
  image:
    repository: linkedin/datahub-kafka-setup
    # tag: "v0.9.6" # defaults to .global.datahub.version
  resources:
    limits:
      cpu: 500m
      memory: 1024Mi
    requests:
      cpu: 300m
      memory: 768Mi
  securityContext:
    runAsUser: null
  podSecurityContext:    
    fsGroup: null
    runAsUser: null
  podAnnotations: {}
  extraEnvs:
    - name: DATAHUB_USAGE_EVENT_NAME
      value: "DataHub_UsageEvent_JSON_v1"
    - name: PLATFORM_EVENT_TOPIC_NAME
      value: "DataHub_PlatformEvent_JSON_v1"
    - name: METADATA_CHANGE_EVENT_NAME
      value: "DataHub_MetadataChangeEvent_JSON_v4"
    - name: METADATA_AUDIT_EVENT_NAME
      value: "DataHub_MetadataAuditEvent_JSON_v4"
    - name: FAILED_METADATA_CHANGE_EVENT_NAME
      value: "DataHub_FailedMetadataChangeEvent_JSON_v4"
    - name: METADATA_CHANGE_PROPOSAL_TOPIC_NAME
      value: "DataHub_MetadataChangeProposal_JSON_v1"
    - name: METADATA_CHANGE_LOG_VERSIONED_TOPIC_NAME
      value: "DataHub_MetadataChangeLog_Versioned_JSON_v1"
    - name: FAILED_METADATA_CHANGE_PROPOSAL_TOPIC_NAME
      value: "DataHub_FailedMetadataChangeProposal_JSON_v1"
    - name: METADATA_CHANGE_LOG_TIMESERIES_TOPIC_NAME
      value: "DataHub_MetadataChangeLog_Timeseries_JSON_v1"

brainy-tent-14503

01/19/2023, 9:22 PM

Can you share the logs for the kafka-setup job? Is there an error when running the setup? Also changing the setup job is only part of it, you’ll need these for the GMS container and the MAE/MCE containers (if running standalone). Also frontend for

DATAHUB_USAGE_EVENT_NAME

, basically the ones listed in each section in the docs.

colossal-smartphone-90274

01/20/2023, 8:53 AM

Yep here are the logs, there doesn't appear to be any errors:

Copy code

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 6.1.4-ccs
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: c9124241a6ff43bc
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1674204429018
/tmp/fifo-ZTSp
will start 1
will start 2
will start 3
will start 4
worker 1 started
worker 3 started
waiting, started 2 of 44
worker 2 started
worker 4 started
sending DataHub_MetadataAuditEvent_JSON_v4 --topic DataHub_MetadataAuditEvent_JSON_v4
sending DataHub_MetadataChangeEvent_JSON_v4 --topic DataHub_MetadataChangeEvent_JSON_v4
sending DataHub_FailedMetadataChangeEvent_JSON_v4 --topic DataHub_FailedMetadataChangeEvent_JSON_v4
sending MetadataChangeLog_Versioned_v1 --topic MetadataChangeLog_Versioned_v1
sending MetadataChangeLog_Timeseries_v1 --config <http://retention.ms|retention.ms>=7776000000 --topic MetadataChangeLog_Timeseries_v1
sending MetadataChangeProposal_v1 --topic MetadataChangeProposal_v1
sending FailedMetadataChangeProposal_v1 --topic FailedMetadataChangeProposal_v1
sending DataHub_PlatformEvent_JSON_v1 --topic DataHub_PlatformEvent_JSON_v1
sending DataHub_UsageEvent_JSON_v1 --topic DataHub_UsageEvent_JSON_v1
1 got work_id=DataHub_MetadataAuditEvent_JSON_v4 topic_args=--topic DataHub_MetadataAuditEvent_JSON_v4
3 got work_id=DataHub_MetadataChangeEvent_JSON_v4 topic_args=--topic DataHub_MetadataChangeEvent_JSON_v4
2 got work_id=DataHub_FailedMetadataChangeEvent_JSON_v4 topic_args=--topic DataHub_FailedMetadataChangeEvent_JSON_v4
4 got work_id=MetadataChangeLog_Versioned_v1 topic_args=--topic MetadataChangeLog_Versioned_v1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
3 got work_id=MetadataChangeLog_Timeseries_v1 topic_args=--config <http://retention.ms|retention.ms>=7776000000 --topic MetadataChangeLog_Timeseries_v1
1 got work_id=MetadataChangeProposal_v1 topic_args=--topic MetadataChangeProposal_v1
Created topic MetadataChangeLog_Versioned_v1.
4 got work_id=FailedMetadataChangeProposal_v1 topic_args=--topic FailedMetadataChangeProposal_v1
2 got work_id=DataHub_PlatformEvent_JSON_v1 topic_args=--topic DataHub_PlatformEvent_JSON_v1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic MetadataChangeLog_Timeseries_v1.
3 got work_id=DataHub_UsageEvent_JSON_v1 topic_args=--topic DataHub_UsageEvent_JSON_v1
2 done working
Created topic FailedMetadataChangeProposal_v1.
Created topic MetadataChangeProposal_v1.
4 done working
1 done working
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
3 done working
Topic Creation Complete.
Completed updating config for topic _schemas.

colossal-smartphone-90274

01/20/2023, 8:54 AM

I have followed the procedure for the other containers but I was just testing with the Kafka set up on its own?

colossal-smartphone-90274

01/20/2023, 1:51 PM

I have managed to find the root of my problem: https://github.com/datahub-project/datahub/blob/master/docker/kafka-setup/kafka-setup.sh (~Lines 100) The 4 files that were being created properly have different names to the ones mentioned in the documentation: Here are the working names with values: - name: METADATA_CHANGE_PROPOSAL_TOPIC value: "DataHub_MetadataChangeProposal_JSON_v1" - name: METADATA_CHANGE_LOG_VERSIONED_TOPIC value: "DataHub_MetadataChangeLog_Versioned_JSON_v1" - name: FAILED_METADATA_CHANGE_PROPOSAL_TOPIC value: "DataHub_FailedMetadataChangeProposal_JSON_v1" - name: METADATA_CHANGE_LOG_TIMESERIES_TOPIC value: "DataHub_MetadataChangeLog_Timeseries_JSON_v1" None of them had the "_NAME" portion on the name

colossal-smartphone-90274

01/20/2023, 1:56 PM

When I get to starting up the DataHub jobs I will check whether or not the names need changing for those jobs as well or if its just applicable to the Kafka job

brainy-tent-14503

01/20/2023, 3:37 PM

Thank you for finding this. I will get a ticket opened to make sure the documentation matches the kafka-setup variables. They should match.

colossal-smartphone-90274

01/20/2023, 3:38 PM

Ok sounds good thanks 🙂

2 Views

Open in Slack

Previous Next