A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

captured_output.txt

cleaned_recipe.yml

Hello. I am getting an error trying to ingest metadata via hive from Databricks. I am including my recipe and the captured output (after cleaning up some identifiers and keys).

My DataHub deployment is in Azure Kubernetes Service and I am running the default deployment that is created via the documented helm commands in the getting started guide.
```helm repo add datahub <https://helm.datahubproject.io/>
helm install prerequisites datahub/datahub-prerequisites
helm install datahub datahub/datahub```
I also created the metadata ingestion source using the DataHub UI.

Any ideas on how to fix this?

Hi, Please remove http:// from host_port section in your recipe.

i.e. simply replace line
```host_port: '<https://adb>-&lt;workspace url id&gt;.<http://azuredatabricks.net:443|azuredatabricks.net:443>'```
with
```host_port: 'adb-&lt;workspace url id&gt;.<http://azuredatabricks.net:443|azuredatabricks.net:443>'```

Hi <@U02G4B6ADL6>
Is this true for all hive sources or just databricks? Maybe we can add a validator in hive source config similar to <https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/superset.py#L63> to remove the `http` part so no one else in community faces the same issue?

Hi <@U027B7R23J4>, I think, this is generally true for all sql (rather sqlalchemy) based sources. We do not include _scheme (http://, hive:// )_ in `host_port` field in config. there is a separate `scheme` field in config which take care of this part. We can update Databricks section in <https://datahubproject.io/docs/metadata-ingestion/source_docs/hive|datahub hive docs> to mention
`host_port: &lt;Databricks Server Hostname&gt;:&lt;Port&gt;`  instead of host_port: `&lt;databricks workspace URL&gt;:443` The details of Databricks Server Hostname and Port are available on <https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html#get-server-hostname-port-http-path-and-jdbc-url|link> referenced in doc.

Thanks for clarifying <@U02G4B6ADL6>
I raised a small PR <https://github.com/linkedin/datahub/pull/4330> so people in community don't face this again. I don't have hive running so cannot test it out. Can you review please if this looks fine to you?

Thanks for the feedback. I will remove the http from `host_port` and give it a try. Updating the docs is a good idea as well

I removed the http from `host_port` and this error no longer occurs.