Hi! Did anyone could make make the Spark-Lineage work on Databricks? I added the dependency on my cluster but I get the following when running a job

Caused by: java.lang.ClassNotFoundException: com.linkedin.datahub.lineage.spark.interceptor.DatahubLineageEmitter

Does anyone know if there is anything special to do when running on databricks?

Hey there! I'm not sure anyone has actually tried to run this on Databricks (yet). Is there a way to ensure that the DataHub JAR is on the classpath in Databricks?

Hey @limited-cricket-18852: we haven't yet published the jars to the central repo. Targeting to get it there this week.

^^^

Will update this thread once we have a public artifact to test with!

indeed, we are building the jar locally and upload it on the Cluster as dependency

@dazzling-judge-80093: do you have any ideas here?

I don’t know how it works on databricks but normally you can specify additional jars with the

--jars

option on

spark-submit

pyspark/etc… -> https://spark.apache.org/docs/latest/submitting-applications.html#advanced-dependency-management

I have been using the UI to submit a spark job on databricks for quick tests, and we have to upload the jar on the cluster directly

I think it should if it is published to maven center but currently it is not as far as I know

looks like it has been published yesterday : https://search.maven.org/artifact/io.acryl/datahub-spark-lineage/0.0.1/jar

@limited-cricket-18852: 0.0.1 might be dead on arrival.. will let you know once we have a lib version that works!

ok I see, thanks!

@limited-cricket-18852:

0.0.2

is here.

<https://repo1.maven.org/maven2/io/acryl/datahub-spark-lineage/0.0.2/>

https://datahubproject.io/docs/spark-lineage/

awesome, thank you ! I still get this

java.lang.ClassNotFoundException: com.linkedin.datahub.lineage.spark.interceptor.DatahubLineageEmitter

But I just noticed that this library supports 2.x while I am using 3.x. I will try to make a build upgrading the spark dependency and see how it goes.

How did you specified the jar? This error message says Spark can’t find the Emitter class which should come from the jar and this error I think is not a spark 2 vs 3 related error.

as specified int he documentation I added the following configuration to my Cluster:

spark.jars.packages                          io.acryl:datahub-spark-lineage:0.0.2
spark.extraListeners                         com.linkedin.datahub.lineage.spark.interceptor.DatahubLineageEmitter
spark.datahub.lineage.mcpEmitter.gmsUrl      <http://localhost:8080>

If this knows to look for the DatahubLineageEmitter, it means that it correctly parsed the

spark.extraListeners

parameter. But it would not find the class either: • setting the configuration

spark.jars.packages

• adding the jar to the job (fat jar) • uploading the jar to the cluster (databricks has a special UI for this)

Do you see any other error during startup or only the class not found exception is what you see?

unfortunately I do not get any error or “strange” log before this stacktrace

@careful-pilot-86309 can you please take a look? ^

Are you using databricks? We have not tested it with databricks yet.

Thank you! I will be off for a couple of weeks, and I will then make more tests.