Hey,
Can we run spark lineage datahub integration ...
# getting-starteds
silly-finland-62382
08/25/2022, 5:45 PMHey,
Can we run spark lineage datahub integration on databricks?
d
dazzling-judge-80093
08/26/2022, 7:56 AMI think it should work, do you see any issue with it?
s
silly-finland-62382
08/26/2022, 8:11 AMit is not working, as I setup the spark lineage code but data is not written to datahub , but its written on databricks instead of datahub
d
dazzling-judge-80093
08/26/2022, 8:12 AMwhat do you mean in
written on databricks instead of datahubs
silly-finland-62382
08/26/2022, 12:59 PMI mean how can we run spark lineage code on databricks cluster
d
dazzling-judge-80093
08/26/2022, 1:01 PMmaybe this can help in this -> https://stackoverflow.com/questions/70974028/addsparklistener-equivalent-in-databricks
s
silly-finland-62382
08/26/2022, 1:20 PMspark listener added successfully
silly-finland-62382
08/26/2022, 1:20 PMbut the thing data is not able to write on datahub using spark lineage ?
d
dazzling-judge-80093
08/26/2022, 1:21 PMI guess you have to let databricks to being able to access your datahub instance on network level
s
silly-finland-62382
08/26/2022, 1:23 PMhow we can do ? databricks team said that this issue at datahub level
d
dazzling-judge-80093
08/26/2022, 1:23 PMWhat issue?
s
silly-finland-62382
08/26/2022, 1:24 PMalso there is no proper doc how we can leverage the datahub with databricks
we set the config properties for dayhub listener and all
after that spark listner is initiated properly but when run
Copy code
df = spark.read.format("csv").option("header", "true").load("dbfs:/FileStore/shared_uploads/nishchay.agarwal@meesho.com/services_classification.csv")
df.write.mode("overwrite").saveAsTable("test_tableq1") this commandit it does not changesimpact on datahubd
dazzling-judge-80093
08/26/2022, 1:24 PMDo you see any error messagess? What do you mean in
data is not able to write on datahub using spark lineagedazzling-judge-80093
08/26/2022, 1:25 PMIn the logs you should see some error message
s
silly-finland-62382
08/26/2022, 1:25 PMthere is no such message
silly-finland-62382
08/26/2022, 1:25 PMinstead of it it writes data on databricks database
d
dazzling-judge-80093
08/26/2022, 1:25 PMCan you see the lineage plugin initialized?
dazzling-judge-80093
08/26/2022, 1:26 PMWhat do you mean it writes data to databricks database? Datahub lineage emitter doesn’t write anything to db
s
silly-finland-62382
08/26/2022, 1:26 PMyeah but emiutter is not running
silly-finland-62382
08/26/2022, 1:26 PMOn Spark context startup
YY/MM/DD HHmmss INFO DatahubSparkListener: DatahubSparkListener initialised.
YY/MM/DD HHmmss INFO SparkContext: Registered listener datahub.spark.DatahubSparkListener
On application start
YY/MM/DD HHmmss INFO DatahubSparkListener: Application started: SparkListenerApplicationStart(YY/MM/DD HHmmss INFO McpEmitter: REST Emitter Configuration: GMS url <rest.server>
silly-finland-62382
08/26/2022, 1:27 PMonly this is showning biut after that emitter message not showing
silly-finland-62382
08/26/2022, 3:49 PMspark.databricks.cluster.profile singleNode
spark.datahub.databricks.cluster datahub_databricks
spark.master local[*, 4]
spark.datahub.rest.token eyJhbGciOiJIUzI1NiJ9.eyJhY3RvclR5cGUiOiJVU0VSIiwiYWN0b3JJZCI6Im5pc2hjaGF5LmFnYXJ3YWwiLCJ0eXBlIjoiUEVSU09OQUwiLCJ2ZXJzaW9uIjoiMiIsImV4cCI6MTY2OTExMDE3MywianRpIjoiMjRiZWM0ZGEtYWQwMC00MTNlLTk5NjEtNmE0ZjMyOTE1OTFmIiwic3ViIjoibmlzaGNoYXkuYWdhcndhbCIsImlzcyI6ImRhdGFodWItbWV0YWRhdGEtc2VydmljZSJ9.7WW2ej0OPq-9SLrWGfiKEHeuf6xiXzFUleaaomet1zc
spark.sql.session.timeZone IST
spark.datahub.rest.server http://172.31.18.133:8080
spark.extraListeners datahub.spark.DatahubSparkListener
fs.s3a.credentialsType AssumeRole
silly-finland-62382
08/26/2022, 3:50 PMWe are using this spark conf for build spark lineage but no lineage I can see on datahub , don't know what happen
silly-finland-62382
08/26/2022, 6:21 PMcan u help on this