Hi, thanks in advance for your help. we installed ...
# ingestion
i
Hi, thanks in advance for your help. we installed DataHub, for exploring as metadata aggregator accompanying IoT datasets. Past my metadata is in the same DB(Mongo) along with data in simple json/json-ld format. How can I ingest the same metadata in json format with my specific information model(key-value sets) into DataHub information model(key-value sets/pairs). Though DataHub is rich with features for exploring meta-data aggregation, browse through search & lineage, I am finding difficult moving forward in moving my data into DataHub.
b
@important-machine-62199 Are you looking for a way to ingest custom metadata into DataHub, instead of using one of the provided source connectors? If so, you can do so using the "File to DataHub" ingestion source and by handcrafting JSON to create Datasets, Dashboards, etc. If you are using Mongo, we do have a prebuilt source connector for that!
i
@big-carpet-38439 thanks for your time in replying to me. I have no prob starting with file-to-datahub ingestion. But, I have my own information model, which like to ingest into Datahub for the purpose of Lineage/Provenance, Linkage, Visualization, Search. I have server farms, would like to track them through metadata on DataHub. Each server is having capabilities(hw, sw, apps & sensors/sub-systems attached on them) with data sets generated/transforms by diff apps(ML, Data engineering). Looking to represent this metadata in a way I could track provenance, linkage(to see correlations from diff datasets/features, etl steps, ACL's). Trying to understand Datahub's information model(allowed keys of the json date to be allowed@present) which i could map/use for my PoC. Also not able to get the (semantics, syntax is ok it's json or avro-json)json files to be written for ingesting into Datahub. I know, asking very dumb questions. I am novice of databases & data engineering development, but skilled in their architecture fitment & performances engineering.