Hi Folks, I'm new to DataHub. When using DataHub S...
# troubleshoot
f
Hi Folks, I'm new to DataHub. When using DataHub Spark Lineage (io.acryldatahub spark lineage0.10.1-1) with a Spark job, it ingests lineage perfectly. However, in the MySQL DataHub database, the "createdby" field shows "urnlicorpuser:__datahub_system". As a result, I cannot remove the lineage manually through the DataHub UI. Could anyone please provide a solution? Additionally, when executing this write command
Copy code
spark.sql("select * from <database>.<table_source>").write.mode("append").format("parquet").saveAsTable("<database>.<table_sink>")
The lineage, as shown in the image below, has been inferred perfectly for the sink table. However, the source table displays the location on my Hadoop Data Lake, even though I'm reading from a table, not a path.
a
Really interesting! Do you think you could share the logs from the spark ingestion job? We may be able to see their what went wrong. the recipe could help too
r
Can you please share the recipe?