Could anyone please help me with exporting and imp...
# ingestionw
witty-keyboard-20400
10/18/2021, 4:45 AMCould anyone please help me with exporting and importing metadata present in DataHub to preferably JSON dump? If that's not possible then export to binary.
@nice-planet-17111 / @big-carpet-38439 any pointer?
👀 1
n
nice-planet-17111
10/18/2021, 6:41 AMI'm really hoping to find a way to export metadata present in Datahub in json format, currently i'm checking GraphQL API, but i'm kinda lost... hehe. If you're looking for a way to import metadata in json file to datahub, you can just use file as data source (if i understood your intention currently)
nice-planet-17111
10/18/2021, 6:41 AMnice-planet-17111
10/18/2021, 6:42 AMHere's the convo we were having about exporting data from datahub https://datahubspace.slack.com/archives/CUMUWQU66/p1634284782168200
w
witty-keyboard-20400
10/18/2021, 10:29 AM@nice-planet-17111 Importing from JSON is straightforward, there is an example for that already.
I'm specifically looking for exporting the entire metadata present in DataHub at any time to JSON/binary dump.
b
better-orange-49102
10/18/2021, 11:55 AMDump MySQL table to file?
b
big-carpet-38439
10/18/2021, 1:04 PMImporting from json is straightforward as you’ve mentioned. As for export we tend to go for RDS dumps to Parquet file format. There’s no formal way to batch dump everything. What are you hoping to achieve with this export feature?
w
witty-keyboard-20400
10/19/2021, 2:12 AM@big-carpet-38439
In a dev environment, I let different data source owners to add tags, desc etc for each field. This needs to be migrated to production deployment.
witty-keyboard-20400
10/19/2021, 2:13 AM@better-orange-49102 Would the auto-gen type of IDs of tables get positioned properly after importing that dump in another MySQL instance?
b
better-orange-49102
10/19/2021, 2:30 AMi didnt try shifting it to another instance, i just did a dump, deleted the db, and recreated it using the dump in the past
13 Views