Hello, were facing some problems with Spark jobs integration. We're creating a scala notebook whose jobs reads from a hive table using

spark.sql()

and writes the data into another hive table. Even though the lineage appears, it has wrong names in the components, in the source it shows the S3 location and in the spark job it shows the method name as you can see in the image bellow. We expected to appear the

schema_name.table_name

instead of the S3 location. Is there something more we can configure? Another question: is the demo source code available somewhere?

@careful-pilot-86309 @hundreds-photographer-13496 ^

We're expecting

test-scala-application

.

Demo code is the master branch of the datahub github repository. Updated every day.

@shy-fireman-88724 Pipeline (dataflow) name is what we pick from application name. Task names are picked up automatically from spark itself. As you know, each spark application contains multiple sql executions. Tasks ( datajob) corresponds to single spark sql execution, hence we dont have specific name to the task. We have tried to be as close as what spark history shows for ease of tracking. Datahub task name corresponds to sql execution description of history server.

@careful-pilot-86309, thank you for your answer and sorry for the delay to reply. We are currently running a DAG in airflow, with the configuration present in the file

airflow-configs.txt

, and we invoke the notebook with a

PapermillSparkOperator

. You can find the notebook code in the file

simple_test.txt

, the result of this DAG is in the image

example-scala.png

. The output of the sql execution in spark is in the file

sql_execution.txt

.

@shy-fireman-88724 Thanks for all the inputs. It really helped to understand the problem at hand. I would like to have one last piece of information. Can you share the debug level enabled logs for the given setup? I am not sure how it works for airflow but you might need to add bellow configuration in your log4j properties file: log4j.logger.datahub.spark=DEBUG log4j.logger.datahub.client.rest=DEBUG

@careful-pilot-86309 I wasn't able to get what you asked. I added the properties that you mentioned to the

log4j.properties

os my spark container (I couldn't do that with our airflow because of the way it is built), but they didn't appear in the output of the logs. What I could find was this warnings logs in datahub containers, they are in the file

logs-datahub.log

. Can you share with us what do you think that is happening in this scenario? Maybe it will help us to solve the problem.

@shy-fireman-88724 These are datahub-gms logs. I will need debug logs from spark execution side.

@careful-pilot-86309 , the spark debug logs: