Dear all, I’m trying to load a data pipeline via python SDK. We use our own tool for pipelines, but it’s similar to Airflow. When I execute it, I get this error: Any clue ? Thanks

File "/opt/homebrew/lib/python3.11/site-packages/datahub/emitter/rest_emitter.py", line 221, in emit
    self.emit_mce(item)
  File "/opt/homebrew/lib/python3.11/site-packages/datahub/emitter/rest_emitter.py", line 233, in emit_mce
    raw_mce_obj = mce.proposedSnapshot.to_obj()
                  ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'proposedSnapshot'

Here we an example how you can create DataFlow/DataJob and emit it. We use the same api in our Airflow plugin -> https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_job_dataflow_new_api_simple.py https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_job_dataflow_new_api_verbose.py

Thanks I’ll take a look at them

Hello @jolly-crayon-48557, have a look in the documentation: https://datahubproject.io/docs/api/tutorials/lineage/#add-column-level-lineage and a more complex example: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_emitter_datajob_finegrained.py

ok, thanks Nikola!