A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi team :smiling_face_with_tear: Is there a way to sink data to file `from datahub`? I want to save it like a snapshot of current status of datahub, and i want it to be the same json format that is used for sink (or data source), so i can customize them and ingest it to datahub later. (Or is there any other way to solve this via mysql DB... etc?)

reading current state from datahub into file? You can go through the APIs and query the data. But it is not as simple currently as the metadata ingestion framework

what kind of customizations are you looking for?

maybe instead of the kafka or rest sink you can use file sink which will give what is ingested into a file

you can use that file as a base for your transformations

Hi <@U027B7R23J4> , thank you for the reply `You can go through the APIs and query the data.-&gt;` Can you elaborate them or is there a doc that i can refer to? What i want to do is,
• I want to update metadata from a data source every day, 
• But I want to keep a separate json file, so i can specifically choose whether i will reflect the change to the datahub system (column-specific, etc.. ) 
• But during the same time, there will be other people editing things via Datahub UI
• So i think i have to fetch current status from datahub to file first (to avoid conflict), then edit the json file, then ingest it to datahub 

<https://datahubproject.io/docs/api/graphql/overview/> <@U02EXJZ0KMM> I have not used it myself so far. But I think team said this is the primary API for querying data

Thank you for sharing, i will check it ! :slightly_smiling_face:

Thanks <@U027B7R23J4>. <@U02EXJZ0KMM>, hopefully your questions are answered!

<@U02EXJZ0KMM>, just FYI - metadata that is added via the UI (user editable info) does not get overridden across ingestion runs.

<@U02AYK6RSPP> That is not true for all cases. Some things do get overwritten by metadata ingestion. You can talk with <@UV0M2EB8Q> for details. He had mentioned that team is looking at cleaning this up over time