Hello, may I ask whether datahub has the capability to ingest lineage information or ERD relationships directly from some sort of file like DBML? I though the json_schema source looked like something for this purpose but I'm not 100% sure.
✅ 1
a
acceptable-morning-73148
07/19/2023, 9:18 AM
You can use the Python SDK for example to ingest custom lineage information. This example https://github.com/datahub-project/datahub/blob/cc2dc342c6164b0a1d332d45acdc68d31b[…]gestion/examples/library/lineage_emitter_dataset_finegrained.py gives a pretty good idea how to do it.
The approach I've followed was:
1. Ingest a sample database into a local DataHub environment using the out of the box PostgreSQL ingestor
2. Look at the example and script lineage between some tables (both table and colum-level lineage)
3. Get accustomed to the logic and how lineage looks like in the UI of DataHub
4. Implement lineage for the custom source I have
5. Test and deploy in a CI/CD pipeline