https://datahubproject.io logo
Join Slack
Powered by
Hi folks, I noticed that a next to no metadata see...
# ingestion
p
Hi folks, I noticed that a next to no metadata seems to be extracted by the Superset extractor apart from Dashboards, Charts and associated Datasets. The Documentation tab is empty for Dashboards and Charts; Properties are empty for Dashboard. And the owner of these object is also not set. Furthermore, it seems that the Description of Charts is not transferred anywhere. ā€¢ Is this the current state of implementation or am I doing anything wrong here? ā€¢ How can I check what is being extracted? Can I see the record that has been transferred to DataHub somewhere? And a bit more specific question: It seems that Superset's Virtual Datasets are not respected nicely. They get represented as (Snowflake) tables but as they do not exist as physical tables/views, we cannot access the underlying SQL code. Where should I file this bug? At datahub-project's or at acryldata's DataHub github repo? Hoping that these many questions are not too overwhelming šŸ™‚
m
Hi @purple-ghost-64569: a feature request on https://feature-requests.datahubproject.io/ would be perfect for the combination of bugs and enhancements that you would like to see.
To see the record that is being transferred to DataHub, you can set the 
sink:
in your recipe to file (https://datahubproject.io/docs/metadata-ingestion/sink_docs/file)
this will write out the events to a file
šŸ‘ 1
m
I have noticed the same and meant to fix those eventually. Never got the time though.... For the virtual datasets, I have debugged the code and noticed there was something missing from the Superset REST API to be able to do a good job. If I recall correctly, the Superset endpoint that Datahub is querying is not returning a boolean indicating if the dataset is virtual or not. In my organization, we have a team dedicated to improve Superset and I have raised this to them, but never did a follow-up.
So the endpoint listing the datasets is returning a list of dataset where each dataset has an attribute named 
kind
, which is set to 
virtual
or 
physical
. But the Datahub source is querying the 
dataset/{id}
endpoint and for some reason, that endpoint is not returning a 
kind
attribute. Another thing that is missing is that the virtual table can (doesn't have to) be derived from a physical table and we would need to parse the SQL to figure that out. We do this for mode already: https://github.com/datahub-project/datahub/blob/611feb95c5bb4c9f9b43591fa44e5b1b8e02a4f0/metadata-ingestion/src/datahub/ingestion/source/mode.py#L451
p
@modern-monitor-81461 Thanks for your reply. Is saw your previous post on this topic: https://datahubspace.slack.com/archives/CUMUWQU66/p1642002202121300?thread_ts=1642001412.121100&cid=CUMUWQU66 On a side note: I noticed that the Superset API apparently returns owner information, but, as it seems, this information is not propagated into DataHub. I do see 
lastModified
information, but no ownership. Do you have an idea if there is a good reason for this? The Superset objects all look so orphaned in DataHub...
2 Views
Open in Slack
PreviousNext
Powered by