Hi team could someone help me clarify a confusion I had abou DataHub #all-things-deployment

Hi team, could someone help me clarify a confusion...

better-lamp-35584

11/09/2023, 11:13 PM

Hi team, could someone help me clarify a confusion I had about UI-ingestion? when defining UI ingestion recipes, I’m using the default sink, which is the REST endpoint, that makes me think metadata get ingested via REST endpoint. However, I know the UI-ingestion is backup by Datahub Actions, and I have an executor running within the Action pod. The event source is Kafka, which is the only supported event source at this moment. I’m confused about the path of metadata get ingested to GMS, is it via REST endpoint or via Kafka in UI-ingestion?

rich-barista-93413

11/16/2023, 9:11 AM

@green-football-43791 might be able to speak to this!

green-football-43791

11/16/2023, 5:43 PM

Hey @better-lamp-35584 - I can definitely help with this!

green-football-43791

11/16/2023, 5:43 PM

are you still struggling with this piece?

better-lamp-35584

11/16/2023, 5:49 PM

Hi @green-football-43791, yes, I would like to learn more about what happened when a UI-ingestion is triggered manually or on schedule, I think this piece of info is missing in datahub docs.

better-lamp-35584

11/16/2023, 5:51 PM

my current understanding is when UI ingestion is triggered, frontend will send messages to

MetadataChangeLog_v1

, the ingestion-executor pipeline subscribe to that topic, once it receives the message, it start processing the ingestion recipe and then start ingestion accordingly, is my understanding correct?

green-football-43791

11/16/2023, 7:11 PM

Hey @better-lamp-35584 would you have a moment to chat now? might be easier over a call

Open in Slack

Previous Next