Hi everyone,
I had tried a couple of tests on spark lineage with hive table.
Here is my result:
• Hive Lineage situation
• spark 2.3.x
◦ orc, text format: both parent, child
◦ parquet format: only child
• spark 2.4.x
◦ text format: both parent, child
◦ orc, parquet format: only child
I wonder that orc, parquet format didn't fully support yet?
Thanks!
📖 1
🔍 1
l
lively-cat-88289
04/17/2023, 7:07 AM
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request.
Let's double check a few things first: ✅ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? ✅ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Did you find a solution to your issue? ❌ Sorry you weren't able to find a solution. I'm sending you some tips on info you can provide to help the community troubleshoot. Whenever you feel your issue is solved, please react ✅ to your original message to let us know!
a
astonishing-answer-96712
04/18/2023, 5:11 PM
hey, what DH version are you running? I don’t think the orc format is fully supported, but it may also be an unsupported spark version
Hi @astonishing-answer-96712
Sorry for missing the message.
My DH version is 0.10.0.6 and I tested all spark 2.3.x and 2.4.x got the same result.
Thanks~