https://datahubproject.io logo
Join Slack
Powered by
Hi everyone, glad to be here....I have couple of q...
# ingestion
b
Hi everyone, glad to be here....I have couple of questions regarding my current setup. As you can see I have Qlik imported and I am currently struggling to get dataset to chart lineage....could use some help if anybody had tried the same with Qlik
l
Hi Roko, I also ingested Qlik dashboards by building mce. I considered .qvd files as datasets and qvw as dashboards. I’m curious to know what you consider as chart in Qlik. Here is an exemple to emit the dataset-to-chart lineage: https://github.com/linkedin/datahub/blob/master/metadata-ingestion/examples/library/lineage_dataset_chart.py
b
Hi Corentin, thanks for reaching out. I consider sheet a dashboard and barcharts, gauge, etc as charts. I am aware of the sample code to emit the dataset to chart lineage. I ran it but I don't see it in the UI.
here's the basic chart to dashboard lineage....what i want now is to add dataset before the chart but can't seem to make it work
I am using Qlik's Engine API to fetch the data
b
And you's using the "inputs" field in Chart model to do this?
Whitespace in urn is not technically illegal, no!
What are the URN of the Chart and the Dataset you are attempting to connect?
And what does the lineage MCPW look like?
b
Hi John, thanks for your response(s).... dataset's urn is 
urn:li:dataset:(urn:li:dataPlatform:qlik,Actual,PROD)
and chart's is 
urn:li:chart:(qlik,dummy)
I have tried both the inputs field method and via
Copy code
input_datasets: List[str] = [
    builder.make_dataset_urn(platform="qlik", name="RegMans", env="PROD"),
    builder.make_dataset_urn(platform="qlik", name="Master_Calendar", env="PROD"),
    builder.make_dataset_urn(platform="qlik", name="Actual", env="PROD"),
    builder.make_dataset_urn(platform="qlik", name="SubscriptionRevenue", env="PROD")
]

last_modified = ChangeAuditStampsClass()

chart_info = ChartInfoClass(
    title="dummy",
    description="",
    lastModified=last_modified,
    inputs=input_datasets
)


chart_info_mcp = MetadataChangeProposalWrapper(
    entityType="chart",
    changeType=ChangeTypeClass.UPSERT,
    entityUrn=builder.make_chart_urn(platform="qlik", name="dummy"),
    aspectName="chartInfo",
    aspect=chart_info,
)
after I run this, chart named dummy appears under Charts/qlik but without any Inputs or lineage
same thing is happening if I try it with looker dataPlatform so I can exclude that it's because of "non-native" Qlik platform
and all of the above datasets actually exist...it's just that I cannot seem to connect the two 🙂
and I managed to emit dataset2dataset lineage
b
Oh wow
Looking at this again now, to see if I can find anything obvious
Quick question - if you fill out the "description" field in the ChartInfo aspect, do you see that reflected into DataHub?
And you can find the chart via search, right?
@blue-boots-43993 I also notice you have salesforce in there -- awesome!!!
Would you be willing to share code for that?
@blue-boots-43993 I just tried to reproduce this locally 😞
its there!!
What version of datahub cli and server are you running?
d
Hey John and Roko, I have the exact same problem. I added Tableau as a source and ingested Metadata through CLI. Anyway I am not able to connect datasets to the charts through the input field. I am running Datahub version 0.8.23.1 and I think it must be related to the newer version. I am also not able to ingest dataset to chart lineage with bootstrap metadata but it was working before (e.g. I just created another looker chart “baz3” and could not connect it to the SampleKafkaDataset).
(would be great if we have a workaround at least because I would like to demo this feature on Tuesday next week to a client that wants to roll-out Datahub)
b
Hi Mike - reading up now
@dazzling-appointment-34954 Which Server version are you on? I think this is perhaps an issue with the previous server version. Also, I'm assuming you are not using Neo4j right?
d
@big-carpet-38439 we just updated to the newest server version on thursday evening, can you point me how to figure out the used version? I am also not sure whether we use Neo4j or not as someone from my Ops team set up the server, I think we just use your provided docker containers in a kubernetes cluster. I am pretty sure it must be related to the current version we use, as the initial ingestion of metadata (full-dump of bootstrap data) worked fine including chart lineage. When I do it now it does not show any lineage, so it seems to be a general problem with this version. I guess we will need to update to a newer version? Do you have any other idea how to make it work at least for the demo tomorrow?
b
Hi @big-carpet-38439...really sorry for this delay in responses...I got sick last week and was off work.
Quick question - if you fill out the "description" field in the ChartInfo aspect, do you see that reflected into DataHub?
positive
I also notice you have salesforce in there -- awesome!!! Would you be willing to share code for that?
Yes I do, and it's nothing spectacular....I downloaded a free trial of CDATA's driver and basically used a standard sqlalchemy recipe
What version of datahub cli and server are you running?
0.8.22.1
these are all the running containers (ran them via quickstart)
i also have a demo tomorrow so this would be hell of a help. Currently my company chose Datahub as our tool to support Data Governance and I need to deliver tomorrow 🙂
I just upgraded both the server and cli to the latest versions....issue persists.... @big-carpet-38439 can you share the code for your reproduced example?
ah nevermind....I got it working!
looking nice so far
d
Thats great to hear Roko, well done ! Would you mind sharing what version of CLI you used? And how did you upgrade it? (sorry for the beginner question)
b
Wonderful!!
@blue-boots-43993 Seems that the containers may have been outdated
@dazzling-appointment-34954 If you pull the latest "head" containers or version 
0.8.24
of DataHub this issue should be resolved.. Sometimes the images are cached by docker and the new version is not fetched, so you need to be sure that you're Docker image is actually being updated
b
@dazzling-appointment-34954 just ran 
datahub dokcer quickstart
in order to fetch new versions of containers and then
Copy code
python -m pip install --upgrade acryl-datahub
to update the python package
d
thanks a lot guys! we updated to the new GMS and cli version and it works 😄 🎉
b
nice to hear
b
Woohoo!!!
Thanks for the great feedback folks 🙂
f
Hello @loud-camera-71352 , we are also trying to ingest Qlik dashboard. would be helpful if you can explain the process and prerequisites (if any) to do so.
b
hi, i'm writing Qlik as a new ingestion source (got carried away with new organization in my company....BUT, I would love someone to test it out with me)
@fresh-cricket-75926 do you want to be the one to help out?
Open in Slack
PreviousNext
Powered by