Hi everyone! I want to analyze some spark sql comm...
# troubleshootr
ripe-electrician-13049
06/17/2022, 3:07 AMHi everyone! I want to analyze some spark sql commands in spark 3. And I upgrade the spark dependency in build.gradle like following:
'sparkSql' : 'org.apache.sparkspark sql 2.123.2.1',
'sparkHive' : 'org.apache.sparkspark hive 2.123.2.1'
But I when I integrate the jar file with my project, I got the error like:
How could I fix this bug?
m
mammoth-bear-12532
06/17/2022, 5:54 AM
@careful-pilot-86309 do you have any idea what might be going on here?
c
careful-pilot-86309
06/17/2022, 6:11 AMWhat is the datahub-spark-lineage version you are using? If its 0.8.35 please upgrade to 0.8.36 and then try
r
ripe-electrician-13049
06/17/2022, 7:07 AMHi @careful-pilot-86309, I just pull the latest code from github. And I want to use spark3, so I change the dependency version. Where to upgrade the datahub-spark-lineage version?
c
careful-pilot-86309
06/17/2022, 8:33 AMNo need for version if using latest code
r
ripe-electrician-13049
06/17/2022, 9:55 AMBut in latest master code, I find that the spark sql dependency is 2.11:2.4.8. But I want to use spark3. So I upgrade the depency. But I find the error which show in the image. How could I use spark 3 in latest master project?
c
careful-pilot-86309
06/17/2022, 10:13 AMno need to update dependency. spark-lineage works with spark 3.2 with current configuration
r
ripe-electrician-13049
06/18/2022, 12:46 AMBut we want to do some development based on source code to support analyzing more spark sql commands. For example, in our spark job. We want to get lineage when we use the class 'BatchScanExec'. This class exists in spark 3.0. So how could we analyze such command?
ripe-electrician-13049
06/20/2022, 6:55 AMHi @mysterious-waiter-64784 , our team want to do secondary development based on the data source. Since our spark job use spark3. We want to parse some spark sqls using datahub to get the lineage. But currently, the master code of datahub use spark2. So some classes like 'BatchScanExec' cannot be parsed. So we want to upgrade the version of the spark in datahub. But I have met the problem in the screenshot. Could you pls help us fix that?
ripe-electrician-13049
06/22/2022, 7:03 AMHi @mysterious-waiter-64784 @careful-pilot-86309, could you pls share some advice?
m
mammoth-bear-12532
06/22/2022, 7:50 AM
Hi @ripe-electrician-13049 the request seems reasonable, we'll get back to you with some suggestions in the next day
mammoth-bear-12532
06/25/2022, 7:40 PM
Hi @ripe-electrician-13049: to support this correctly, we'll have to split up the spark module into spark2 and spark3 sub-modules with different build.gradle in each
r
ripe-electrician-13049
06/27/2022, 3:01 AMHi @mysterious-waiter-64784, that will be great! So what's your plan to do the split and when could I get the splited spark modules in mater branch?
ripe-electrician-13049
01/11/2023, 1:34 AM@mysterious-waiter-64784, I pulled the latest code and find out that the project still use spark2.0. When will you share the code to support spark3.0?