Hi Estelle, first of all thanks for your suggestion. I double checked 3 different PowerBI projects, the service principals I'm using have Admin rights of the workspace but the problem is always the same: "_Pipeline finished successfully; produced 0 events in #.## seconds._" Below the yaml file I'm using: ---------------------------------------- _$ cat ./pbi_436_dev.yaml_ source: type: "powerbi" config: # Your Power BI tenant identifier _tenant_id: ######################_ # Ingest elements of below PowerBi Workspace into Datahub _workspace_id_pattern:_ allow: - ###################### # Azure AD Application identifier _client_id: ######################_ # Azure AD App client secret _client_secret: ######################_ # Enable / Disable ingestion of user information for dashboards _extract_ownership: true_ # Enable/Disable extracting workspace information to DataHub containers _extract_workspaces_to_containers: True_ # Enable / Disable ingestion of endorsements. # Please notice that this may overwrite any existing tags defined to ingested entities! _extract_endorsements_to_tags: false_ _# dataset_type_mapping is fixed mapping of Power BI datasources type to equivalent Datahub "data platform" dataset_ _dataset_type_mapping:_ _# dataset_type_mapping is fixed mapping of Power BI datasources type to equivalent Datahub "data platform" dataset_ _dataset_type_mapping:_ Sql: mssql sink: type: datahub-rest config: server: 'http://localhost:8080' ---------------------------------------- Pipeline finished successfully; produced 0 events in #.## seconds. $ datahub version DataHub CLI version: 0.9.6.2 Python version: 3.9.2 (default, Feb 28 2021, 170344) [GCC 10.2.1 20210110] I'm trying to collect more details with the command: "_datahub --debug ingest -c ./pbi_436_dev.yaml -n_" Is there a way to increase the verbosity? Is there someone that can debug/fix the issue? Thanks in advance for your support!

Hi everyone, Somebody could help me I already configured the power bi service with the security group, but this keeps happening (editado)

We are getting "bad request" while ingesting PowerBI report, it seems it processing all reports but getting stuck at one of them... here is the debug log. Workspace is added correctly. Can anyone please shed some light?

Hello everyone! The question is about ingesting data from PowerBI. Current implementation only ingests data from cloud PowerBI. Is there a way to ingest metadata from a locally deployed PowerBI (from on-premises server)?

Hi, ingestion works fine, but I do not see any lineage at all. I do get a lot of warnings when ingesting like this: _'Installation.Equipment_korttekster': ['Function calls not found', 'Function calls not found'],_ I'm running 0.10.2, has anyone seem something similar?

Hi again, when trying to add additional config options I keep getting this error: Failed to configure the source (powerbi): 1 validation error for PowerBiDashboardSourceConfig _workspace_id_as_urn_part_ _extra fields not permitted (type=value_error.extra)_ same for almost all the config I try. My recipe looks like this: # Ingest elements of below PowerBi Workspace into Datahub workspace_id_pattern: allow: - 8d85743b-432e-4c87-9ec8-9ec08f94bec0 - 63c88901-7861-4736-8144-044cf179d296 # deny: dataset_type_mapping: PostgreSql: postgres Oracle: oracle Sql: mssql #ingest using admin api admin_apis_only: true # Enable / Disable ingestion of ownership information for dashboards extract_ownership: true # Enable / Disable ingestion of lineage information extract_lineage: true # Enable/Disable extracting workspace information to DataHub containers extract_workspaces_to_containers: true # Enable / Disable ingestion of endorsements. # Please notice that this may overwrite any existing tags defined to ingested entities! extract_endorsements_to_tags: false workspace_id_as_urn_part: true If I remove or comment out the last line everything runs okay.

Hi @flaky-librarian-65126 You can find valid property config in documentation at https://datahubproject.io/docs/generated/ingestion/sources/powerbi#config-details

@gentle-hamburger-31302 yes, and there it shows the workspace_id_as_urn_part , so wondering why it does not work

Hi, just wanted to check that someone has successfully been able to get lineage working with Power BI data, e.g. App->Report->Dataset->Source

Okay, so our datahub is running v0.10.4. My Datahub CLI version is 0.10.4 and Python is 3.11.4. Running an ingest of my yaml file (which has worked fine in other version setups gives me: Command failed: Failed to find a registered source for type powerbi: mutable default <class 'datahub.metadata.schema_classes.NullTypeClass'> for field datahubDataType is not allowed: use default_factory Downgraded python to 3.10 and it now runs, so looks like 3.11 is not working.

Hi all 🙂 I am having a few issues when running the Powerbi ingestion on a large project containing ±60 workspaces (the data I managed to extract so far is 85MB): 1 - Some workspaces ingestion crash with

404

. This happens when datahub tries to access some dashboard tiles using the admin

tiles

endpoint. I think this is a bug on PowerBI's side since these dashboards do exist, and other endpoints seem to work fine for them, but I think datahub shouldnt crash in such a case. 2 - overall the ingestion is painfully slow (more than a day) and some of the really large workspaces which have many input sources, never complete ingestion. I tried to debug this a little and it seems that the code is stuck on the

m_query

parsing in the

lark

library. Did anyone else try to use the Powerbi ingestion for large projects?

Hi, I'm trying to get a regx pattern to work for only ingesting Power BI workspaces with a certain string in the name. Can anyone provide the correct syntax? I've tried: workspace_id_pattern: allow: -'.*EBI' but it does not return any of the workspaces that end with that.......using '.*' returns all workspaces....

Hi, I'm trying to ingest a PowerBI workspace using the DataHub UI. When I run the ingestion with admin API disabled the ingestion result is empty, see logs and recipe. When I run the ingestion with admin API enabled, the ingestion seems to ingest everything except charts in reports, and (therefore?) lineage is unavailable between PowerBI datasets and reports. Lineage between snowflake sources and PowerBI datasets is working. I have followed the setup guide, https://datahubproject.io/docs/quick-ingestion-guides/powerbi/setup

Hi, I have previously configured the Power BI report server ingestion and scheduled it for daily ingestion, and it was working fine. But now I'm getting the following error. Can someone help me, please?

has renamed the channel from "integration-powerbi-datahub" to "integrate-powerbi-datahub"

Hello, is there a way to ignore PowerBI report objects/charts when ingesting using the Powerbi module and instead map report lineage directly to Datasets? Thanks in advance.

Hey everyone, I'm attempting to create a recipe to connect to PowerBI but am receiving the following error: "Failed to generate the PowerBi access token. Please check input configuration." Using a debug tool I was able to see that while attempting to obtain an access token the connector received the error "Invalid client secret provided." I verified that my tenant and client IDs are correct and made sure the value for my client secret is correct as well. I'm confused about what could be causing this error. Has anyone else has run into a bizarre issue like this? Thanks.

Hi, Team: I need your assistance for power BI ingestion with Datasets that are related to the report. We have established the pipeline with "include Lineage" function on UI. We also have registered permissions on the power bi side with Admin API admin settings - Microsoft Fabric We're expecting to see the data lineage for the dataset but it appears only the name of power bi report itself. We also checked if the API account is in the workspace in Power BI but it is in there. Would you please verify and give me some advise how we can fix it?

We are testing ingestion of Power BI. We are seeing an issue where all ingested datasets are being placed into one random workspace. Have tried deleting all content and ingesting again several times, same problem. Has anyone else experienced this? We are running 0.11.0, Fairly sure this is related to the two settings: extract_datasets_to_containers: true extract_workspaces_to_containers: true so it happens when both are set to true, the last one being default, but makes no difference if it is explicitly set or not. Setting extract_workspaces_to_containers to false throws an error on my system, but that could/would be an okay way to fix it if it worked. (the error is 'Mapper' object has no attribute 'workspace_key'). Only ingesting workspaces works as tables are placed correctly, the issue is when enabling the dataset_to_container which we really need as the Power BI dataset is the analytical asset we want (would be fine if the tables could be excluded actually). Has anyone else successfully ingested both workspaces and datasets? Could be a Python version issue perhaps, so if you have please let me know what versions you are running.

Hi, Some of our PowerBI dataset tables are missing lineage from their Snowflake source database, presumably because the dataset tables use a parameter to reference the database in the m-query, see attached image. Is there a workaround for this?

Uploaded the incorrect image and edit mode doesn't let me add a new one. Here is the correct one

I’m getting errors like this trying to ingest the powerbi lineage:

[2023-11-23 14:17:26,530] INFO     {datahub.ingestion.source.powerbi.m_query.parser:77} - Unsupported m-query expression for table APR_Visuals.Comparative_(simplified): No terminal matches 'm' in the current parser context, at line 2 col 215

flate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type t
                                        ^
Expected one of: 
	* MORETHAN
	* AS
	* LBRACE
	* RPAR
	* STAR
	* NEWLINE
	* SLASH
	* __ANON_28
	* EQUAL
	* AMPERSAND
	* WS_INLINE
	* OR
	* IS
	* PLUS
	* __ANON_26
	* UNDERSCORE
	* LESSTHAN
	* AND
	* __ANON_27
	* COMMA

Any ideas what might be causing this? does Datahub support DAX parsing?

Hi all! I would like to get some help:-) I am currently trying to ingest metadata and lineage from Power BI, and it works as expected with dashboards and charts (I can see lineage PowerBI Dataset table -> Chart -> Dashboard), however only the charts that are used in Dashboards are ingested; as for reports, the chart section is empty and obviously it lacks lineage as well. What could cause that issue?

I include the recipe:

pipeline_name: power_bi_test_fakhrutdinov_2
datahub_api:
    server: '<http://datahub-datahub-gms.datahub.svc.cluster.local:8080>'
source:
    type: powerbi
    config:
        tenant_id: ***
        client_id: ***
        client_secret: '${PowerBI_AdminApi_R}'
        stateful_ingestion:
            enabled: true
        workspace_id_pattern:
            allow:
                - ***
        admin_apis_only: true
        extract_endorsements_to_tags: true
        extract_ownership: true
        extract_lineage: true
sink:
    type: datahub-kafka
    config:
        connection:
            bootstrap: '***'
            schema_registry_url: '***'
            producer_config:
                security.protocol: SASL_SSL
                sasl.mechanism: SCRAM-SHA-512
                sasl.username: datahub
                sasl.password: '***'
                ssl.ca.location: ***
            schema_registry_config:
                <http://basic.auth.user.info|basic.auth.user.info>: '***'
                ssl.ca.location: ***
        topic_routes:
            MetadataChangeEvent: datahub.MetadataChangeEvent_v4
            MetadataChangeProposal: datahub.MetadataChangeProposal_v1

Hi guys! Is it possible to create an upstream lineage to Power BI Dataset Table from the external source? For instance, I have a Hive table that serves as a source for Power BI Dataset table; if Hive table was created in Datahub, should I see a lineage from it to PBI Dataset table when I ingest data from PowerBI? Or the only possible way to add this type of lineage is to do it manually? Many thanks!

Hi all! I have a question concerning concept mapping between Power BI and Datahub, namely Pages (PBI) <-> Charts (Datahub). Am I assuming correctly, that every page of paginated report should be created in Datahub as a separate Chart? Because it is not the behaviour that we observe when ingesting paginated reports - we get only one Report in DH. Many thanks!

Hi everyone, I'm trying to ingest using a powerbi-report-server recipe and right now I'm getting a succeeded output with 0 ingestions. The logs are showing an warning message saying "failed to fetch reports/linked_reports/powerbi_reports..." with https status-code 401. The user I'm using in the recipe has admin permissions in the localhost reportserver portal. Am I missing something else? Any idea? Thanks in advance.

Hi, using the Allow/Deny pattern in the Power BI ingestion yaml file it seems like the ability to use regex patterns applies to the GUID of the workspace being ingested. Cant really see any use of this and would have expected it to apply to the workspace name. Anyone else with a similar experience?

I posted similar issue here before but just want to make sure if my understanding for my issue is correct. We would like to have the lineage between power BI chart and Amazon Redshift data-sources that is being consumed. The ingestion through UI with activating "include lineage" doesn't work out as expected. My environment is data-on-premise gate way is on between power BI and Redshift data-sources so I assume this would possibly cause some error for retrieving the lineage. Manual ingestion way on Python is worked but I would like to make sure if there is any way to meet this requirements with automatation. Thank you for your help in advance.

Hi, is there a way to exclude each chart being ingested? I only want report/dashboard level ingested. Cheers