https://datahubproject.io logo
Join Slack
Powered by
Hi Team, We are ingesting metadata from Hive platf...
# ingestion
s
Hi Team, We are ingesting metadata from Hive platform using sql alchemy plugin. But it's not picking up the column descriptions of the tables. We are on the version v0.8.43. Can anyone help on this?
g
That’s not supposed to happen! Are you using the 
hive
source or the 
presto-on-hive
source?
s
We are using hive
h
Hi @stocky-truck-96371 can you share recipe used for ingesting hive metadata ? What 
scheme
are you using ?
s
Hi, It's the recipe we are using
Copy code
source:
    type: sqlalchemy
    config:
        platform: "hive"
        include_views: true
        env: DEV
        platform_instance: EDHUB
        connect_uri: sink:
  type: "datahub-rest"
  config:
    server: '<http://datahub-datahub-gms:8080>'
h
It looks like you are using generic sqlalchemy source , can you try using hive source instead ? - You can use 
sqlalchemy_uri
instead of `connect_uri`in hive source Meanwhile, could you share value of 
connect_uri
, especially the part before 
://
s
Copy code
connect_uri: "impala
h
oh, that explains missing descriptions. Are you using impyla package to support impala dialect ?
s
yes
Hi, We have tried the recipe by using hive as source. But it's still not picking up the column description. Recipe, source: type: hive config: include_views: true host_port: env: DEV platform_instance: EDHUB sqlalchemy_uri: "impala://host:port/default?use_ssl=true&auth_mechanism=GSSAPI&kerberos_service_name=hive" sink: type: "datahub-rest" config: server: 'http://datahub-datahub-gms:8080' token:
g
Yes, unfortunately we don’t support column descriptions from impala yet. Please file a feature request https://feature-requests.datahubproject.io/ and we’ll get to it as soon as we can
s
Sure, thanks
2 Views
Open in Slack
PreviousNext
Powered by