<!here> I have tried to ingest the metadata from a...
# ingestion
w
<!here> I have tried to ingest the metadata from a mysql table to datahub and it was ingested successfully but I am doubting it because I am not able to see any data there. I am just able to see what columns are there and the data types (basically the values that are there in header of the SQL table but not the rows that are there). Also if I try to search any data that is there in a row for particular column, I get No data results in UI. I am just wondering whether DataHub provides this feature to search data there or it just ingests the table, columns, column data type or description only. Can anyone help me ?
Here are some screenshots of the table that I have ingested, Metadata that I am able to see in DataHub and the search results page with data thats there in the table
b
If you want to search the data, you'll be better off extending products like CKAN. Datahub deals with metadata, and doesn't store actual data of the table
w
Thanks @better-orange-49102 It will be helpful if you can tell me about some of the tools that we can use for this use case. Also can CKAN be integrated with DataHub or it’s a separate tool that we can use for our use case to store both the metadata and actual data?
b
CKAN is an older data catalog product. it stores data in its own instance. however, it only accepts flat files and cannot read metadata off a database (hence i say need to extend CKAN yourself). And no, it doesnt work with datahub, though maybe devs would be interested in federating catalogs in the future.
thankyou 1
w
Also is there any option to upload some sample data and manage the access control in Datahub, we are just in exploration phase
b
i suppose u could extract a sample of the data from the DB and put it as a link in the documentation "tab".
1
w
Thanks @better-orange-49102 However just to clarify about our use case, we are actually having relational metadata stored in MySQL but not able to get the actual column names You can refer to my screenshots to understand our use case
Is there any organization if you are aware of who has solved similar use case of storing relational metadata in DataHub?
b
but you stored the relational metadata as a table. data hub can retrieve the metadata about the table, but not the data in the table.
sry, not familiar
w
Okay so here is one more QQ, How we should store the relational metadata except tables/views?
Or let me rephrase my question @better-orange-49102, Can you tell me what should be the format of the ingested source be in order to get the column names?
b
im not sure why there is a need to store (metadata of a table) in a table... datahub can read the schema of these tables just fine.
m
Hi @wooden-sunset-90925 from looking at the screenshot it seems like the MySQL table actually contains metadata about other views tables etc.
Such as column names etc
We normally recommend just pointing Datahub ingestion to those raw tables directly to pull out the schemas
However it seems like you have additional business metadata also that is being kept inside this table which you probably want to ingest into Datahub.
For that you will need to write a custom connector that ETLs this metadata and writes to Datahub (using the python sdk)
If you would like to talk to someone to figure out how to accomplish that, happy to set up the call.
b
oh @mammoth-bear-12532 does datahub informs the user of relationship between tables? say primary keys and foreign keys? been meaning to ask about that
w
@mammoth-bear-12532 Thanks for clarification! we exactly have the same use case and would like to setup the call with you if its possible next week on Monday?
b
@better-orange-49102: Our models support it, but I don't think our UI currently shows anything for it.. Definitely something we will built out for at least relational tables.. but some work remaining before that