Hi All, Request team to help with PR https://github.com/datahub-project/datahub/pull/9412 - This is basically addressing broken Spark lineage for hive tables. Message links on other channels posted for the same https://datahubspace.slack.com/archives/C04BSKCPXCK/p1702033468493729?thread_ts=1696582266.105799&cid=C04BSKCPXCK.

@gray-shoe-75895 can you take a look at this one?

We're working on a new spark plugin which we're looking to get out in the next few weeks. cc @dazzling-judge-80093 - does it make sense to review/accept this PR given that?

@gray-shoe-75895 - Meantime would you be able to help us understand what are the capabilities of new spark plugin? In fact we had few things planned in coming quarters, insight into new plugin would help us to reprioritise and align our road map accordingly?

Which capabilities are you looking for? The new plugin will support the latest version of spark, wider spark support, column level lineage, path spec support, etc…

@dazzling-judge-80093 do you think its worth it checking out the current way of lineage with spark or shall we wait until the new plugin is released?

@dazzling-judge-80093 - We were looking for thin client version so that we could avoid deployment of spark DH client jar on our hadoop cluster for every upgrade of spark DH client. Further we would like to see the above issue #9412 is resolved as its breaking spark lineage in our DH.