A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi there! I am using the Hive+Databricks ingest Source plugin, but it doesn’t extract the columns comments. Am I doing something wrong?

Hi <@U02L8V7UDDZ>, column description is one of the very elementary features we support for hive. When you issue a "describe table" statement, do you see the description?

image.png

Hi <@U02AYK6RSPP> thanks for your answer. Yes I get the description when I do a “describe table”. Here is an example from 2 columns from one of our tables:

and here is the same columns when ingesting into a File Sink (easier to see what it built):

And finally, here is the recipe I used:
```source:
  type: hive
  config:
    host_port: <http://XXXXX.cloud.databricks.com:443|XXXXX.cloud.databricks.com:443>
    username: token
    password: XXXXXXXXXXXXXXXXXXXX
    scheme: 'databricks+pyhive'
    database: silver
    options:
      connect_args:
        http_path: 'sql/protocolv1/o/XXXXX/XXXXXXXXXXXXXXX'
sink:
  type: file
  config:
    filename: ./output.json```

Thanks a lot <@U02L8V7UDDZ>. I need to look at if databrick hive instance behave any differently.