A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

image (6).png

image (7).png

image (8).png

Hi,

I think there is a problem *ingesting table column comments using the DataBricks Hive metastore and Hive ingestor*. If I use the delta lake ingestion it works but I can see the comments are in json format, not sure if this is due to delta lake ingestion or a data bricks standard.

Anyone knows if maybe changing any parameter in the recipe it would works with the hive ingestion?

This is the recipe for hive + databricks

```source:
  type: hive
  config:
    host_port: <http://adb-4911762757593392.12.azuredatabricks.net:443|adb-4911762757593392.12.azuredatabricks.net:443>
    username: token
    password: dapi5d4d8bbae4a81e8b4d52b28dd2c94390-2
    scheme: 'databricks+pyhive'
    database: silver
    platform: delta-lake
    profiling:
      enabled: true
    table_pattern:
      allow:
        - silver.calendar_calendars

    options:
      connect_args:
        http_path: 'sql/protocolv1/o/4911762757593392/0426-111319-t6tf09gf'

sink:
  type: datahub-rest
  config: ...............```

Hi <@U02NSLNFFFA> thanks for sharing this, looks like a small bug fix needed on the connector for this.

We already merged the fix for this. Can you try it out with `acryl-datahub[hive]==0.8.41rc2`

but incremental crawling not getting comments

nope sorry <@UV0M2EB8Q> not working on datahub 0.8.41

Are you setting up ingestion in the UI or using the cli to ingest?

```source:
  type: hive
  config:
    host_port: <http://adb-4911762757593392.12.azuredatabricks.net:443|adb-4911762757593392.12.azuredatabricks.net:443>
    username: token
    password: dapi5d4d8bbae4a81e8b4d52b28dd2c94390-2
    scheme: 'databricks+pyhive'
    database: silver
    platform: delta-lake
    table_pattern:
      allow:
        - silver.calendar_calendars

    options:
      connect_args:
        http_path: 'sql/protocolv1/o/4911762757593392/0426-111319-t6tf09gf'

sink:
  type: datahub-rest
  config:
    server:```

*➜ crawler* datahub --version
acryl-datahub, version 0.8.41rc2

<@U02HE6R3F5L> could you check on this one? 

<@U02NSLNFFFA> Fix is not present in your datahub version. please upgrade to atleast 0.8.41.2rc1 ( this is current latest version)