A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi community :wave:  Want some advice. We're using a mixture of Databricks and Redshift in our team. Redshift is connected as an external catalog in Databricks.

This means that one redshift table can show up as a Redshift entity and a Databricks entity. Can I merge them somehow?

My other option is to only ingest the Redshift tables via one source plugin, but:
• If I ingest via Databricks plugin, I lose out on a lot of the usage stats and SQL generated lineage within Redshift
• If I ingest via Redshift plugin, the downstream databricks tables which use the Redshift tables don't show any lineage to the redshift tables.

I'm all sorted now <@U059RL34LLB>, I used <https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/cli/specific/dataset_cli.py#L83-L105> as reference :slightly_smiling_face:

I noticed the cli command overwrites existing siblings. Here's a PR to preserve them: <https://github.com/datahub-project/datahub/pull/9907>