A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hey everyone, I am still very new to the datahub community but stumbled across the following question: We work with a couple of technologies that are not supported (yet), e.g. Talend as a Pipeline tool, Databricks or also SAP R3. Is there a generic way to ingest metadata for those into the catalog or is the only way to build new individual connectors? Thanks in advance for some feedback :slightly_smiling_face:

You need some way to get data out of those tools. If you have a way to get that data out then you can simply use the rest API to send the data to the relevant endpoints in datahub GMS.

But contributing connectors means you can get feedback and collaborate with others who are using similar tools.

That will make your technical debt less over time

e.g. Glue connector was written by a company. some other companies while using the Glue connector found some bugs for which they sent PRs. The original company also benefited from that.

Totally agree, thanks for the quick response Aseem! So that means the Rest-API also exposes some kind of “generic” datasource which I can use to feed my data (which I obviously need to extract from the source system)?
I am totally with you on building connectors rather than just using the API. We already built a nice JDBC based connector for SAP, which we might be able to reuse and contribute. I guess the way forward here is to create a plugin and a recipe?

This is a PR where I added MariaDB as a source <https://github.com/linkedin/datahub/commit/ebe0b3f7c1283cad876d4e8a8ba9fbdb768146be>. It was a small PR given I just needed to sub-type Mysql source. This will give you an idea about the places where you would need to make changes when adding a new source.

If you are working on a PR and face confusion you can post in <#C017W0NTZHR|contribute> channel and people would be happy to help

I haven’t tried yet (very soon!), but looks like Databricks’ Hive is already supported, right? <https://datahubproject.io/docs/metadata-ingestion/source_docs/hive>

Ah right, that slipped through from my side. Thanks for the hint!

<@U02L5TSDD7D> <https://datahubproject.io/docs/metadata-ingestion/adding-source/|Here is a guide> for developing new sources. In a nutshell, the connector will have some configuration and some code to map the 3rd party system models into DataHub metadata models (available as a strongly-typed library in Python). An example source that extracts users and groups from Okta and sinks them to DataHub is shown <https://github.com/linkedin/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/identity/okta.py|here>!