Hi, Team: I need your assistance for power BI ingestion with Datasets that are related to the report. We have established the pipeline with "include Lineage" function on UI. We also have registered permissions on the power bi side with Admin API admin settings - Microsoft Fabric We're expecting to see the data lineage for the dataset but it appears only the name of power bi report itself. We also checked if the API account is in the workspace in Power BI but it is in there. Would you please verify and give me some advise how we can fix it?

I just checked ours and we are seeing the same issue. We have PowerBI being fed data from our Databricks UC but Datahub only shows the lineage up to the Query in PowerBI and doesn't connect to the source tables. I guess i could manually add them but kinda defeats the purpose of automated data cataloging.

@adorable-salesclerk-90917 Thank you for your info. It's very similar situation. In our case, we're ingesting the data from our redshift to power BI. @gentle-hamburger-31302 @gray-shoe-75895 If you could give us some insights for these situation, that would be great.

we see the same on our setup, lineage works between reports, pages and tables, but not to anything else.

Just dug into Help and saw the following:

PowerBI Source supports M-Query expression for below listed PowerBI Data Sources
Snowflake
Oracle
PostgreSQL
Microsoft SQL Server
Google BigQuery
Native SQL query parsing is supported for Snowflake and Amazon Redshift data-sources.

Yep, aware of that, perhaps our source being Azure Synapse is an issue, although in Power BI its the same connector

We have other dashboards that are powered from Neo4j that we are already planning to manually register so we it looks like we will have to do that for everything

@gentle-hamburger-31302 could you look into this? How can we make it easy for folks to register/contribute additional powerbi data source types beyond the ones we've built out ourselves?

One way is to document where to add things in code to get additional sources working

@gentle-hamburger-31302 @gray-shoe-75895 Thank you for the message. Is there any plan to improve the power BI connector that can extract the data lineage into Datahub? We would like to eliminate the manual work as much as possible, so I would like to know. Thank you!

We are going to create a python script to hit the REST API on the same schedule as our powerbi ingestion run to 'manually' connect the pbi product to source from the m_query. Part of our work flow will be for the data analyst to inform data engineers of source and name of dashboard then they will add the script to etl..kinda a workaround but should work ok

Thank you. Understood your solution and that make sense for me. Seems like this is the most feasible solution at this moment.

ill share the code when we have it working if you are interested

Yes, I'm so interested in your proposal. Could you please share?? Thank you so much!!

@stocky-helmet-13192 are you hosting datahub locally or through acryl?

@adorable-salesclerk-90917 Thank you for your response. We're hosting datahub locally.\

here is a python script of helper function one of my dev wrote when we were messing the with local datahub. Hope this helps. We haven't connected out lineage yet as we have other priorities for our next few sprints.

@adorable-salesclerk-90917 Thank you for sharing the code!! I appreciate a lot

@stocky-helmet-13192 the error about "extra field not permitted" likely means that your need to update your version of acryl-datahub