Do we support Hive 1.2 ingestion? Looks like curre...
# ingestion
m
Do we support Hive 1.2 ingestion? Looks like current hive ingestion used pyhive which is only compatible with Hive 2
b
I don't think anyone has built a Hive 1.x connector yet 😕
thank you 1
m
Maybe we can build it. All we need is to connect to hive metastore and get table schemas
l
Would be great to get that contribution @mysterious-lamp-91034! Let us know how we can help.
m
@loud-island-88694 Thanks! I definitely will have some questions in the future. Then I will ask you
l
awesome - also, please let us know if this is blocking your POC of DataHub. We can look into prioritizing this
m
Thanks @loud-island-88694 This is my first question, I followed https://datahubproject.io/docs/metadata-ingestion/adding-source and started from step 4) I updated pip dependencies of plugins and add my hive1x in entry_points https://gist.github.com/wizardxz/8ec1bf9aa94e416a24c856192f45351f#file-gistfile1-txt-L284 But when I run
datahub check plugins
I don't see my hive1x source. Is the build required? Is there a way to debug new source without build? Thanks
g
Could you try rerunning
pip install -e '.'
in the metadata-ingestion directory’s venv?
m
trying
It works. Thanks @gray-shoe-75895
Quick question: Does datahub pull from hive metastore or push from hive metastore? Thanks!
m
@mysterious-lamp-91034 the "datahub ingest" command will pull from hive server. We just implemented a spark-lineage emitter that pushes metadata out from a spark job. A hive emitter would need to be built that does something similar. We don't have it currently, but might be pretty easy to build by following the spark-lineage template.
thank you 1
It will likely just be an implementation of a
HiveHook
thank you 1