Hi guys, I've tried to setup Spark lineage for my ...
# ingestion
p
Hi guys, I've tried to setup Spark lineage for my local datahub instance but it does not seem to be generating the pipeline. Can anyone help me with it.
Copy code
spark = SparkSession.builder()
        .appName("test-application")
        .config("spark.master", "<spark://spark-master:7077>")
        .config("spark.jars.packages","io.acryl:datahub-spark-lineage:0.8.23")
        .config("spark.extraListeners", "datahub.spark.DatahubSparkListener")
        .config("spark.datahub.rest.server", "<http://localhost:8080>")
        .enableHiveSupport()
        .getOrCreate();
this is what I setup in my spark config file.
c
@polite-application-51650 Can you use latest version 0.8.31? Also how are you doing spark-submit? Are you getting any errors?