Hi. Regarding the python ingestion framework, would it make sense to introduce a step between the source and sink where you could attach generic transformation of the events? My use-case right now is to be able to modify dataset names. I want to avoid writing custom sources or sinks for just this small reason and if there was a middle step in a pipeline I could just plug in my name-transformer and be done. WDYT?

👍 We’d be looking at enriching the to-be-ingested data with ownership information from external sources.

@gray-shoe-75895 Thoughts?

Yup. That has always been the plan. A chain of convertors (or transformers) to support enrichment of the events.

@mammoth-bear-12532 We could make a feature contribution for this if you wish.

Sounds good! Let’s agree on the api-s before hand.

Yep we definitely want to do this. In the initial designs we had been thinking about the "Extractor" serving this function, but I'm not totally sure that's the right interface. Chainable converters/transformers is probably closer to what we actually want.