https://datahubproject.io logo
Join Slack
Powered by
Hi All, We’re just trying to configure our datahub...
# troubleshoot
a
Hi All, We’re just trying to configure our datahub instance to work with our production kafka cluster, and we’re seeing some configuration issues when we try to change the topic names so they fit with our naming convention. Specifically changes to the topic 
MetadataChangeProposal_v1
don’t seem to be respected in the gms process. Is this a known problem? I’ve tried poking through the code but can’t immediately see the source
our env vars are configured like so:
Copy code
- name: METADATA_CHANGE_EVENT_NAME
          value: datahub-prod-metadata-change-event
        - name: METADATA_AUDIT_EVENT_NAME
          value: datahub-prod-metadata-audit-event
        - name: FAILED_METADATA_CHANGE_EVENT_NAME
          value: datahub-prod-failed-metadata-change-event
        - name: FAILED_METADATA_CHANGE_PROPOSAL_TOPIC
          value: datahub-prod-failed-metadata-change-proposal
        - name: DATAHUB_USAGE_EVENT_NAME
          value: datahub-prod-data-usage-event
        - name: METADATA_CHANGE_LOG_VERSIONED_TOPIC
          value: datahub-prod-metadata-change-versioned
        - name: METADATA_CHANGE_LOG_TIMESERIES_TOPIC
          value: datahub-prod-metadata-change-timeseries
        - name: METADATA_CHANGE_PROPOSAL_TOPIC
          value: datahub-prod-metadata-change-proposal
        - name: KAFKA_MCE_TOPIC_NAME
          value: datahub-prod-metadata-change-event
        - name: KAFKA_FMCE_TOPIC_NAME
          value: datahub-prod-failed-metadata-change-event
        - name: KAFKA_TOPIC_NAME
          value: datahub-prod-metadata-audit-event
e
@adamant-pharmacist-61996 did this work for you?
a
so to configure the names we ended up with these env vars:
Copy code
- name: METADATA_CHANGE_EVENT_NAME
          value: datahub-prod-metadata-change-event-v4
        - name: METADATA_AUDIT_EVENT_NAME
          value: datahub-prod-metadata-audit-event-v4
        - name: FAILED_METADATA_CHANGE_EVENT_NAME
          value: datahub-prod-failed-metadata-change-event-v4
        - name: METADATA_CHANGE_PROPOSAL_TOPIC_NAME
          value: datahub-prod-metadata-change-proposal-v1
        - name: METADATA_CHANGE_LOG_TOPIC_NAME
          value: datahub-prod-metadata-change-log-versioned-v1
        - name: METADATA_CHANGE_LOG_VERSIONED_TOPIC_NAME
          value: datahub-prod-metadata-change-log-versioned-v1
        - name: METADATA_CHANGE_LOG_LIMITED_TOPIC_NAME
          value: datahub-prod-metadata-change-log-timeseries-v1
        - name: METADATA_CHANGE_LOG_TIMESERIES_TOPIC_NAME
          value: datahub-prod-metadata-change-log-timeseries-v1
        - name: FAILED_METADATA_CHANGE_PROPOSAL_TOPIC_NAME
          value: datahub-prod-failed-metadata-change-proposal-v1
        - name: KAFKA_MCE_TOPIC_NAME
          value: datahub-prod-metadata-change-event-v4
        - name: KAFKA_FMCE_TOPIC_NAME
          value: datahub-prod-failed-metadata-change-event-v4
        - name: KAFKA_TOPIC_NAME
          value: datahub-prod-metadata-audit-event-v4
        - name: DATAHUB_USAGE_EVENT_NAME
          value: datahub-prod-data-hub-usage-event-v1
there seems to be a bit of inconsistency with the naming conventions
potentially some of these are redundant but, for now, its seemingly working
thanks for checking back @early-lamp-41924
e
Yes. This was caused by the difference in env var between gms and mae/mce consumers which we wanted to fix for the new topics.
In the future, we are planning to move to just use these new topics and deprecate the old!
a
yeah, we found out because access controls through kafka started denying creation of the default named topics 🙂
3 Views
Open in Slack
PreviousNext
Powered by