Hi Team, would like to know if its possible to create custom pipelines in Datahub & ingest data against it. We have lot of scheduler & system which executes application/queries which modified tables. we would want to capture all of those jobs in DataHub Lineage

Hi! Yes you can certainly do this! We recommend building custom ingestion sources for this purpose, which fetch metadata from your custom sources, map it datahub’s data model, and then write it into DataHub. Thé sources that DataHub currently has made available (written in python) would be able to serve as an example

Hi @big-carpet-38439, Thanks for getting back on this. Can we create custom pipelines other than Airflow & Spark ? We have Control-m & other data movement products which moves data around (between & within) system. We want to have pipeline for all of these schedulers & data movement product

This sample code shows emitting lineage edges: https://datahubproject.io/docs/lineage/sample_code And here's where we emit "Pipelines" and "Jobs" that are found inside of Airflow: https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub_provider/_lineage_core.py#L225

I have already looked at this sample code, i believe they are to add airflow task but i want to create new pipeline (Control-M) like airflow & push lineage

Yeah so you'd need to use the library to emit lineage from Control M!

Cool, i am finding it difficult to understand how to emit lineage for new pipeline (Control M). Should we create this pipeline first and then push lineage ? should i look at https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub_provider/_lineage_core.py#L225 ?

@orange-night-91387 Would you mind working with @eager-oxygen-76249 on this? Ryan has been thinking of how we can simplify the experience of integrating with DataHub, specifically around emitting custom metadata. He can hopefully point you in the right direction. We can also start a separate slack channel if it would help

Hi! For emitting MCPWrappers from a new source the current approach is following the example of what has been done in the existing source emitters. This involves interacting with the source APIs to pull relevant metadata out and plugging that into an MCPWrapper object to emit. We provide utility functions in the lineage_core.py file that make creating these objects easier. To pull the metadata out of the source you will have to do something similar to how we pull lineage from other sources like: https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/dbt.py#L802 Or similar to our Airflow hook integration which pulls from annotated inlets & outlets on each DAG: https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub_provider/lineage/datahub.py#L69

Awesome John & Ryan ! i will check these, try to ingest create custom pipeline, ingest lineage & get back to you