Hi everyone! how are you?! I created a spark inges...
# getting-startedk
kind-angle-49107
08/29/2023, 8:26 PMHi everyone! how are you?! I created a spark ingestion on databricks, the pipeline appears on the UI with the right startedAt time, but with 0 tasks… What it could be? I setted spark.sql.legacy.createHiveTableByDefault to false, my application do: df.write.mode(‘overwrite’).saveAsTable(tablename, path=pathname)
kind-angle-49107
08/29/2023, 8:27 PMOn log4j logs DataHubSparkListener apperars on only on SQL Started execution event
a
aloof-gpu-11378
08/30/2023, 5:35 AM@famous-waitress-64616 might be able to speak to this!
f
famous-waitress-64616
08/30/2023, 3:48 PMHi Bruno, I'm doing alright, hbu? Hopefully we can get this resolved for you soon! Unfortunately I'm not familiar with our spark source, @dazzling-judge-80093 can you take a look here?
k
kind-angle-49107
08/30/2023, 3:55 PM@delightful-ram-75848 @famous-waitress-64616 @dazzling-judge-80093
Hey Guys! thank you for the help! I have some more data that should help you:
I’m running a DBR 10.4 with Spark 3.2.1 Job on a Standalone cluster via Databricks API. The job is running using Spark Python Task
My Datahub version is: 0.10.4.2
Spark lineage jar is: 0.10.5-6rc1
Spark confs:
extraListeners
rest server
remove partition pattern = true (tested true and false with same result)
coalesce jobs = true (with this config set to true i was able to get a task on the UI with the same name as the app name but without any dataset or lineage info)
databricks cluster
Pipeline always have Started At and Completed At, but without any lineage information
kind-angle-49107
08/30/2023, 3:56 PMI already tested with spark 3.1.2 and jar 0.10.5 same result
kind-angle-49107
08/30/2023, 4:00 PMAll my tables are hive external tables , physical files are stored on Azure Data lake storage, but it is mounted on databricks dbfs
kind-angle-49107
08/30/2023, 4:01 PMthis is my spark application, very simple
kind-angle-49107
08/30/2023, 4:02 PMAlso, I tried to set up debugging logs but the datahub documentation is for log4j and my databricks cluster is using log4j2, is it possible to configure it?
kind-angle-49107
08/30/2023, 4:05 PMall tables on my spark application are delta
kind-angle-49107
08/30/2023, 10:41 PMUpdate:
I tried updating my jar to 0.10.5-6rc2
Tested with different spark versions and applications using saveAsTable, save , sparkSQL without success…
Same result, Pipeline shows on the UI with Start time, when I use spark.stop() at the end it shows Completed At data on the UI also, but databricks jobs fails since it’s not supposed to use stop on databricks. But always without any data about lineage or tasks…
Does someone has any idea what to do more? thanks!!
kind-angle-49107
08/31/2023, 12:38 PM@dazzling-judge-80093
Those are the only logs datahub appears here (But debug logs are disabled) :
23/08/30 192342 INFO SparkContext: Registered listener datahub.spark.DatahubSparkListener
23/08/30 192429 INFO AsyncEventQueue: Process of event SparkListenerSQLExecutionStart(executionId=0, ...) by listener DatahubSparkListener took 7.124870504s.
kind-angle-49107
08/31/2023, 12:47 PMThere are no McpEmitter INFO logs
a
aloof-gpu-11378
08/31/2023, 2:26 PMUnfortunately it is a known and yet unsolved issue that Databricks doesn’t call theevent.onApplicationEnd
We will need to do further investigation into why this is happening there and what we can do about it.
k
kind-angle-49107
08/31/2023, 2:27 PMGot it, thanks @dazzling-judge-80093 , this impact lineage info also?
kind-angle-49107
08/31/2023, 2:28 PMscan and write tables
d
dazzling-judge-80093
08/31/2023, 2:30 PMI think you should have some lineage but not full lineage as on SparkListenerSQLExecutionEnd event it should send in -> https://github.com/datahub-project/datahub/blob/e7d140f82df393b98821f8a49e74ae3995[…]k-lineage/src/main/java/datahub/spark/DatahubSparkListener.java
dazzling-judge-80093
08/31/2023, 2:31 PMjust for a quick test, please can you try to write out in a non-delta format (like parquet or json)?
k
kind-angle-49107
08/31/2023, 2:32 PMSure, i'll give it a try! brb, thank you
kind-angle-49107
08/31/2023, 2:41 PM@dazzling-judge-80093 same output 😞 pipeline appears on the UI, with StartedAt and 0 tasks
kind-angle-49107
08/31/2023, 2:41 PMsame logs as before
kind-angle-49107
08/31/2023, 2:46 PMthis is my query plan
kind-angle-49107
08/31/2023, 2:51 PMmy code does: spark.table(table).filter(condition).select(col1, col2).write.mode('overwrite').format('parquet').save(path)
kind-angle-49107
08/31/2023, 3:19 PM@dazzling-judge-80093 It appeared a different task now!!! Niiicee!! But with no lineage to hive yet, do you know what it could be? the task is linked by table name? path?
kind-angle-49107
08/31/2023, 3:21 PMimage.png,image.png
kind-angle-49107
08/31/2023, 3:58 PMParquet appears task without Lineage, Delta does not appear task
Parquet: InsertHadoopFsRelationCommand
kind-angle-49107
08/31/2023, 3:59 PMDelta: SaveIntoDataSourceCommand
kind-angle-49107
08/31/2023, 4:09 PMmaybe I should include platform instance? i didn't understand what it is
kind-angle-49107
08/31/2023, 4:21 PMPath on task appears without dbfs:/ and on hive it appears with dbfs:/ scheme on location
kind-angle-49107
08/31/2023, 10:35 PMUpdate:
Lineage worked for downstream hive tables!!! I had to write it as parquet and use saveAsTable method from Spark.
I’ll try more to get it done for delta tables. Just one question, spark integration ingest upstream hive tables too? @dazzling-judge-80093