https://datahubproject.io logo
Join Slack
Powered by
Hey, I'm having an issue with configuring topic na...
# troubleshoot
b
Hey, I'm having an issue with configuring topic names. I have the following env vars set in my GMS containers:
Copy code
❯ kubectl exec -n xxxxxx datahub-datahub-gms-7bfb87d7cd-7sksf -- env | grep METADATA                                                                                
METADATA_CHANGE_EVENT_NAME=xxx.MetadataChangeEvent_v4
METADATA_AUDIT_EVENT_NAME=xxx.MetadataAuditEvent_v4
FAILED_METADATA_CHANGE_EVENT_NAME=xxx.FailedMetadataChangeEvent
However, GMS startup fails with:
Copy code
java.lang.IllegalStateException: Topic(s) [MetadataChangeEvent_v4] is/are not present and missingTopicsFatal is true
I also tried setting 
SPRING_KAFKA_LISTENER_MISSING_TOPICS_FATAL=false
but that didn't seem to do anything. Any advice?
e
Hey. Unfortunately, there are a few more env that you need to set because the env names for mae/mce consumers are different. Here is the full list
Copy code
- name: METADATA_CHANGE_EVENT_NAME
              value: {{ .metadata_change_event_name }}
            - name: FAILED_METADATA_CHANGE_EVENT_NAME
              value: {{ .failed_metadata_change_event_name }}
            - name: METADATA_AUDIT_EVENT_NAME
              value: {{ .metadata_audit_event_name }}
            - name: KAFKA_TOPIC_NAME
              value: {{ .metadata_audit_event_name }}
            - name: KAFKA_MCE_TOPIC_NAME
              value: {{ .metadata_change_event_name }}
            - name: KAFKA_FMCE_TOPIC_NAME
              value: {{ .failed_metadata_change_event_name }}
            - name: DATAHUB_USAGE_EVENT_NAME
              value: {{ .datahub_usage_event_name }}
            - name: METADATA_CHANGE_PROPOSAL_TOPIC_NAME
              value: {{ .metadata_change_proposal_topic_name }}
            - name: FAILED_METADATA_CHANGE_PROPOSAL_TOPIC_NAME
              value: {{ .failed_metadata_change_proposal_topic_name }}
            - name: METADATA_CHANGE_LOG_VERSIONED_TOPIC_NAME
              value: {{ .metadata_change_log_versioned_topic_name }}
            - name: METADATA_CHANGE_LOG_TIMESERIES_TOPIC_NAME
              value: {{ .metadata_change_log_timeseries_topic_name }}
b
Hey, thanks for responding. all of these need to be set in the datahub-gms process's environment?
e
Yeah. you can see taht a few envs are tied together
Copy code
METADATA_CHANGE_EVENT_NAME
and KAFKA_MCE_TOPIC_NAME for example
we combined mce/mae consumers and gms together and they had different env variables unfortunately
for the new topics that we will be migrating to mainly MCP, MCL_versioned, MCL_timeseries, we only have one envs
b
we combined mce/mae consumers and gms together
when did this happen? in the helm charts they're separated
where did that code sample you posted come from? it looks like a helm template but I don't see it in 
acryldata/datahub-helm
. Just trying to make sure I understand, so is this documentation and this example out of date?
sorry for alll the q's, just a bit confused
thanks again for your help
e
ah it’s from our internal charts
So we tried to simplify moving parts. If this field is set to true, https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/values.yaml#L63 it will spawn a separate mae/mce consumer pod
👍 1
By default this is false, meaning that these processes are spawned within gms pod
Ah yes. we need to add more variables to that doc, thanks for pointing it out!
but basically, you need to add all the envs in that doc (not just the ones for gms) as well as the topic names for the new topics!
b
gotcha, makes sense now, thanks!
3 Views
Open in Slack
PreviousNext
Powered by