Hi there, My usecase is to capture the data lineage from the Spark jobs which run using KubernetedPodOpearator in Airflow. is this integration with Airflow being supported in Datahub. I am a newbie to Datahub. Any help is appreciated! Thnks

@gray-shoe-75895 probably has more thoughts here.

Yes we've heard of some others who successfully emitted lineage information from Spark, so it's definitely possible - the spark agent seems like the way to go

Thanks for the response! I am trying to load data from postgresql to s3 after applying some transformations in my Spark job ( which run in k8s, via Airflow -KubernetesPodOperator). The dataset lineage with the job information and the transformation details needs to be pushed to Datahub. Can you please suggest how to model the lineage events.

@gray-shoe-75895 ^ can you please provide pointers to existing usages of Lineage event and how to use the emitter API?

I have used Spline agent to extract lineage information from Spline agent to Datahub. For @brave-appointment-76997, I think there are two problems. one is that Airflow's operator lineage information. this has been discussed and proposed as a new entity called Jobs. Secondly it's the Spark lineage. Spark lineage is more about the relationship among datasets and its columns.

Hey @brave-appointment-76997 - sorry I missed this message. I have a PR up right now that should simplify this process - emitting lineage from Python will look like this (once the PR is merged): https://github.com/linkedin/datahub/blob/5b10691e97b21b776718a416b66219766ab9a7bd/metadata-ingestion/examples/library/lineage_emitter.py

@loud-island-88694 Yes, I have installed datahub. We are exploring the features of datahub and current usecase in plan is getting the lineage of datasets which undergoes certain transformation and stored back to S3 via spark job. Datasources currently dealt with are Kafka,Postgres etc.. and destination is S3. The spark jobs are being executed in k8s using kubernetespod operator.

Currently, the majority of the sources only support datasets/descriptions/schemas/etc. We have Airflow operators available that make it easier to declare and emit lineage. For Spark, we don't have anything prebuilt, but it shouldn't be too hard to emit using the Spline agent

@gray-shoe-75895 by Airflow operators, are you referring the link https://github.com/linkedin/datahub/blob/5b10691e97b21b776718a416b66219766ab9a7bd/metadata-ingestion/examples/airflow/lineage_emission_dag.py

Yep

okay..thanks for the response. is there a kafka emitter example (I really could not find one in Github)

Sure, I just made one: https://github.com/linkedin/datahub/blob/6f427dcedeaeb248fb089b5a78f99888fa8ade3e/metadata-ingestion/examples/library/lineage_emitter_kafka.py

Thank you for this example