Hi everyone! We are trying out Data Hub’s push-based ingestion using Python REST emitter and we have some questions. We’ve created a process that each time it runs it ingests multiple Metadata Change Proposal events essentially describing a Data product: GlossaryTerm associated to GlossaryNode, several Datasets/DataJobs with Owners and Domain, Dataset’s Schema Fields and relationships between assets. What we’ve tested out so far is defining all MetadataChangeProposalWrapper events needed and then emitting each of them sequentially using

DatahubRestEmitter

’s

emit()

method. Is there a better way to do this? How should we deal with the emission of multiple events as if it was some sort of single transaction and the possible rollback in case of any error with the ingestion process? Thanks in advance! 🙂

Very interesting! One of the strategies you can use today is attach the same run-id to all the events using the systemMetadata field

That sounds like a good strategy, we will give it a try. Thanks a lot!