Hi Team,
I want to acheive column level lineage using Pyspark script in Datahub. Created Pyspark script to read data from mysql server (datahub docker img) using JDBC driver, performed some transformation and load it in the same location. I also ingested Mysql server in datahub Ref:
https://datahubproject.io/docs/api/tutorials/lineage/#add-lineage (Add Column-level Lineage). My Task is reflecting correctly on spark platform in datahub but unable to achieve column level lineage. can some one please guide me on the same?
version:
datahub:v0.12.1
spark:v3.5.0