Hi team, I'm new to datahub. Tried to use spark lineage feature, but I met some problems. Listener counld find the downstream but couldn't find the upstream! I had already set "spark.datahub.coalesce_jobs" to true. I would be grateful if someone could give me some clue, thank you! Below is my sample code: find downstream Persons1, Persons2 can't find upstream Persons on datahub UI lineage

val ds1 = spark.read
        .format("jdbc")
        .option("driver","com.mysql.cj.jdbc.Driver")
        .option("url", "jdbc:mysql://{myhost}:3306/xxx")
        .option("dbtable", "Persons")
        .option("user", "xxx")
        .option("password", "xxx")
        .load()


ds1.write.mode(SaveMode.Append)
.format("jdbc")
.option("driver","com.mysql.cj.jdbc.Driver")
.option("url", "jdbc:mysql://{myhost}:3306/xxx")
.option("dbtable", "Persons1")
.option("user", "xxx")
.option("password", "xxx")
.save()

ds1.write.mode(SaveMode.Append)
.format("jdbc")
.option("driver","com.mysql.cj.jdbc.Driver")
.option("url", "jdbc:mysql://{myhost}:3306/xxx")
.option("dbtable", "Persons2")
.option("user", "xxx")
.option("password", "xxx")
.save()

Hi @rapid-zoo-88437 Could you please share your DataHub screenshots and also logs ? You can find log detail at https://datahubproject.io/docs/metadata-integration/java/spark-lineage#debugging

Hi @gentle-hamburger-31302 Thanks for your response! Here is my logs and screemshots:

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See <http://logging.apache.org/log4j/1.2/faq.html#noconfig> for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
23/03/27 18:03:13 INFO SparkContext: Running Spark version 2.3.0
23/03/27 18:03:13 INFO SparkContext: Submitted application: Nozzleusage-test-mysql
23/03/27 18:03:14 INFO SecurityManager: Changing view acls to: hdfs,hive
23/03/27 18:03:14 INFO SecurityManager: Changing modify acls to: hdfs,hive
23/03/27 18:03:14 INFO SecurityManager: Changing view acls groups to: 
23/03/27 18:03:14 INFO SecurityManager: Changing modify acls groups to: 
23/03/27 18:03:14 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hdfs, hive); groups with view permissions: Set(); users  with modify permissions: Set(hdfs, hive); groups with modify permissions: Set()
23/03/27 18:03:15 INFO Utils: Successfully started service 'sparkDriver' on port 63062.
23/03/27 18:03:15 INFO SparkEnv: Registering MapOutputTracker
23/03/27 18:03:15 INFO SparkEnv: Registering BlockManagerMaster
23/03/27 18:03:15 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/03/27 18:03:15 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/03/27 18:03:15 INFO DiskBlockManager: Created local directory at C:\Users\leeandy\AppData\Local\Temp\blockmgr-a3f69039-43d2-4241-9bf4-3d0889be2a60
23/03/27 18:03:15 INFO MemoryStore: MemoryStore started with capacity 1963.2 MB
23/03/27 18:03:15 INFO SparkEnv: Registering OutputCommitCoordinator
23/03/27 18:03:15 INFO Utils: Successfully started service 'SparkUI' on port 4040.
23/03/27 18:03:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at <http://host.docker.internal:4040>
23/03/27 18:03:15 INFO Executor: Starting executor ID driver on host localhost
23/03/27 18:03:15 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 63103.
23/03/27 18:03:15 INFO NettyBlockTransferService: Server created on host.docker.internal:63103
23/03/27 18:03:15 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
23/03/27 18:03:15 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, host.docker.internal, 63103, None)
23/03/27 18:03:15 INFO BlockManagerMasterEndpoint: Registering block manager host.docker.internal:63103 with 1963.2 MB RAM, BlockManagerId(driver, host.docker.internal, 63103, None)
23/03/27 18:03:15 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, host.docker.internal, 63103, None)
23/03/27 18:03:15 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, host.docker.internal, 63103, None)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
23/03/27 18:03:15 INFO SparkContext: Registered listener datahub.spark.DatahubSparkListener
23/03/27 18:03:15 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/C:/Users/leeandy/Desktop/work_space/nozzleusage/spark-warehouse').
23/03/27 18:03:15 INFO SharedState: Warehouse path is 'file:/C:/Users/leeandy/Desktop/work_space/nozzleusage/spark-warehouse'.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See <http://www.slf4j.org/codes.html#StaticLoggerBinder> for further details.
23/03/27 18:03:16 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
23/03/27 18:03:17 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
23/03/27 18:03:17 INFO metastore: Trying to connect to metastore with URI <thrift://linxta-pecdp00:9083>
23/03/27 18:03:17 INFO JniBasedUnixGroupsMapping: Error getting groups for hive: Unknown error.
23/03/27 18:03:17 WARN UserGroupInformation: No groups available for user hive
23/03/27 18:03:17 INFO metastore: Connected to metastore.
23/03/27 18:03:18 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
23/03/27 18:03:18 INFO SessionState: Created local directory: C:/Users/leeandy/AppData/Local/Temp/d47d8971-457e-499a-8d24-15a609c24d2d_resources
23/03/27 18:03:18 INFO SessionState: Created HDFS directory: /tmp/hive/hive/d47d8971-457e-499a-8d24-15a609c24d2d
23/03/27 18:03:18 INFO SessionState: Created local directory: C:/Users/leeandy/AppData/Local/Temp/hdfs/d47d8971-457e-499a-8d24-15a609c24d2d
23/03/27 18:03:18 INFO SessionState: Created HDFS directory: /tmp/hive/hive/d47d8971-457e-499a-8d24-15a609c24d2d/_tmp_space.db
23/03/27 18:03:18 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.2) is file:/C:/Users/leeandy/Desktop/work_space/nozzleusage/spark-warehouse
23/03/27 18:03:20 INFO CodeGenerator: Code generated in 179.0849 ms
23/03/27 18:03:20 INFO SparkContext: Starting job: save at HiveToMysqlDriver.scala:168
23/03/27 18:03:20 INFO DAGScheduler: Got job 0 (save at HiveToMysqlDriver.scala:168) with 1 output partitions
23/03/27 18:03:20 INFO DAGScheduler: Final stage: ResultStage 0 (save at HiveToMysqlDriver.scala:168)
23/03/27 18:03:20 INFO DAGScheduler: Parents of final stage: List()
23/03/27 18:03:20 INFO DAGScheduler: Missing parents: List()
23/03/27 18:03:20 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at save at HiveToMysqlDriver.scala:168), which has no missing parents
23/03/27 18:03:20 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 12.3 KB, free 1963.2 MB)
23/03/27 18:03:20 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 6.2 KB, free 1963.2 MB)
23/03/27 18:03:20 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on host.docker.internal:63103 (size: 6.2 KB, free: 1963.2 MB)
23/03/27 18:03:20 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1039
23/03/27 18:03:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at save at HiveToMysqlDriver.scala:168) (first 15 tasks are for partitions Vector(0))
23/03/27 18:03:20 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
23/03/27 18:03:20 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, executor driver, partition 0, PROCESS_LOCAL, 7677 bytes)
23/03/27 18:03:20 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
23/03/27 18:03:20 INFO CodeGenerator: Code generated in 20.9835 ms
23/03/27 18:03:20 INFO JDBCRDD: closed connection
23/03/27 18:03:20 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1107 bytes result sent to driver
23/03/27 18:03:20 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 236 ms on localhost (executor driver) (1/1)
23/03/27 18:03:20 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
23/03/27 18:03:20 INFO DAGScheduler: ResultStage 0 (save at HiveToMysqlDriver.scala:168) finished in 0.500 s
23/03/27 18:03:20 INFO DAGScheduler: Job 0 finished: save at HiveToMysqlDriver.scala:168, took 0.556526 s
23/03/27 18:03:20 INFO SparkContext: Starting job: save at HiveToMysqlDriver.scala:177
23/03/27 18:03:20 INFO DAGScheduler: Got job 1 (save at HiveToMysqlDriver.scala:177) with 1 output partitions
23/03/27 18:03:20 INFO DAGScheduler: Final stage: ResultStage 1 (save at HiveToMysqlDriver.scala:177)
23/03/27 18:03:20 INFO DAGScheduler: Parents of final stage: List()
23/03/27 18:03:20 INFO DAGScheduler: Missing parents: List()
23/03/27 18:03:20 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[8] at save at HiveToMysqlDriver.scala:177), which has no missing parents
23/03/27 18:03:20 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 12.3 KB, free 1963.2 MB)
23/03/27 18:03:20 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 6.2 KB, free 1963.2 MB)
23/03/27 18:03:20 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on host.docker.internal:63103 (size: 6.2 KB, free: 1963.2 MB)
23/03/27 18:03:20 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039
23/03/27 18:03:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[8] at save at HiveToMysqlDriver.scala:177) (first 15 tasks are for partitions Vector(0))
23/03/27 18:03:20 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
23/03/27 18:03:20 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, executor driver, partition 0, PROCESS_LOCAL, 7677 bytes)
23/03/27 18:03:20 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
23/03/27 18:03:20 INFO JDBCRDD: closed connection
23/03/27 18:03:20 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1021 bytes result sent to driver
23/03/27 18:03:20 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 116 ms on localhost (executor driver) (1/1)
23/03/27 18:03:20 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
23/03/27 18:03:20 INFO DAGScheduler: ResultStage 1 (save at HiveToMysqlDriver.scala:177) finished in 0.130 s
23/03/27 18:03:20 INFO DAGScheduler: Job 1 finished: save at HiveToMysqlDriver.scala:177, took 0.134558 s
23/03/27 18:03:21 INFO SparkContext: Invoking stop() from shutdown hook
23/03/27 18:03:21 INFO SparkUI: Stopped Spark web UI at <http://host.docker.internal:4040>
23/03/27 18:03:21 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/03/27 18:03:21 INFO MemoryStore: MemoryStore cleared
23/03/27 18:03:21 INFO BlockManager: BlockManager stopped
23/03/27 18:03:21 INFO BlockManagerMaster: BlockManagerMaster stopped
23/03/27 18:03:21 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/03/27 18:03:21 INFO SparkContext: Successfully stopped SparkContext
23/03/27 18:03:21 INFO ShutdownHookManager: Shutdown hook called
23/03/27 18:03:21 INFO ShutdownHookManager: Deleting directory C:\Users\leeandy\AppData\Local\Temp\spark-561a2662-5b42-4dd9-ac47-195c4657d363

Process finished with exit code 0

Hello, @rapid-zoo-88437 Were you able to solve it ? Having the same problem here

Hi @important-bear-9390, I updated the spark version to 2.4.x and MySql worked. And I also tested the hive situation Hope this helps FYI Hive Lineage • spark 2.3.x ◦ orc, text format: both parent, child ◦ parquet format: only child • spark 2.4.x ◦ text format: both parent, child ◦ orc, parquet format: only child