Hi all! I need some help :slightly_frowning_face: ...
# integrate-powerbi-datahub
s
Hi all! I need some help 🙁 Has anyone faced any issues while ingesting metadata from powerBI? Here are the details of my attempt:
Copy code
source:
    type: powerbi
    config:
        scan_timeout: 60000
        extract_reports: true
        tenant_id: <tenant_id>
        workspace_id_pattern:
            allow:
                - '.*'
        env: DEV
        client_id: <client_id>
        client_secret: '********'
        extract_ownership: true
        dataset_type_mapping:
            Snowflake: snowflake
sink:
    type: datahub-rest
    config:
        server: '<http://host:8080>'
I get no errors but the result of the ingestion is that I don't have anything in my tenant. Here is the log output:
Copy code
~~~~ Execution Summary ~~~~

RUN_INGEST - {'errors': [],
 'exec_id': 'd2313617-911c-4a97-8d25-0679f414c08e',
 'infos': ['2023-01-24 14:28:14.879955 [exec_id=d2313617-911c-4a97-8d25-0679f414c08e] INFO: Starting execution for task with name=RUN_INGEST',
           '2023-01-24 14:28:20.932698 [exec_id=d2313617-911c-4a97-8d25-0679f414c08e] INFO: stdout=Obtaining venv creation lock...\n'
           'Acquired venv creation lock\n'
           'venv setup time = 0\n'
           'This version of datahub supports report-to functionality\n'
           'datahub  ingest run -c /tmp/datahub/ingest/d2313617-911c-4a97-8d25-0679f414c08e/recipe.yml --report-to '
           '/tmp/datahub/ingest/d2313617-911c-4a97-8d25-0679f414c08e/ingestion_report.json\n'
           '[2023-01-24 14:28:17,021] INFO     {datahub.cli.ingest_cli:165} - DataHub CLI version: 0.9.6\n'
           '[2023-01-24 14:28:17,075] INFO     {datahub.ingestion.run.pipeline:179} - Sink configured successfully. DataHubRestEmitter: configured '
           'to talk to <http://host:8080>\n'
           '/tmp/datahub/ingest/venv-powerbi-0.9.6/lib/python3.10/site-packages/datahub/ingestion/source/powerbi/powerbi.py:740: '
           'ConfigurationWarning: env is deprecated and will be removed in a future release. Please use platform_instance instead.\n'
           '  config = PowerBiDashboardSourceConfig.parse_obj(config_dict)\n'
           '[2023-01-24 14:28:17,295] INFO     {datahub.ingestion.source.powerbi.proxy:211} - Trying to connect to '
           '<https://login.microsoftonline.com/tenant_id>\n'
           '[2023-01-24 14:28:17,295] INFO     {datahub.ingestion.source.powerbi.proxy:326} - Generating PowerBi access token\n'
           '[2023-01-24 14:28:17,382] INFO     {datahub.ingestion.source.powerbi.proxy:340} - Generated PowerBi access token\n'
           '[2023-01-24 14:28:17,383] INFO     {datahub.ingestion.source.powerbi.proxy:213} - Able to connect to '
           '<https://login.microsoftonline.com/tenant_id>\n'
           '[2023-01-24 14:28:17,522] INFO     {datahub.ingestion.source.powerbi.proxy:211} - Trying to connect to '
           '<https://login.microsoftonline.com/tenant_id>\n'
           '[2023-01-24 14:28:17,522] INFO     {datahub.ingestion.source.powerbi.proxy:326} - Generating PowerBi access token\n'
           '[2023-01-24 14:28:17,608] INFO     {datahub.ingestion.source.powerbi.proxy:340} - Generated PowerBi access token\n'
           '[2023-01-24 14:28:17,608] INFO     {datahub.ingestion.source.powerbi.proxy:213} - Able to connect to '
           '<https://login.microsoftonline.com/tenant_id>\n'
           '[2023-01-24 14:28:17,609] INFO     {datahub.ingestion.run.pipeline:196} - Source configured successfully.\n'
           '[2023-01-24 14:28:17,610] INFO     {datahub.cli.ingest_cli:120} - Starting metadata ingestion\n'
           '[2023-01-24 14:28:17,611] INFO     {datahub.ingestion.source.powerbi.powerbi:765} - PowerBi plugin execution is started\n'
           '[2023-01-24 14:28:17,611] INFO     {datahub.ingestion.source.powerbi.proxy:690} - Request to get groups endpoint '
           'URL=<https://api.powerbi.com/v1.0/myorg/groups>\n'
           '[2023-01-24 14:28:17,989] INFO     {datahub.ingestion.reporting.file_reporter:52} - Wrote SUCCESS report successfully to '
           "<_io.TextIOWrapper name='/tmp/datahub/ingest/d2313617-911c-4a97-8d25-0679f414c08e/ingestion_report.json' mode='w' encoding='UTF-8'>\n"
           '[2023-01-24 14:28:17,989] INFO     {datahub.cli.ingest_cli:133} - Finished metadata ingestion\n'
           '\n'
           'Cli report:\n'
           "{'cli_version': '0.9.6',\n"
           " 'cli_entry_location': '/tmp/datahub/ingest/venv-powerbi-0.9.6/lib/python3.10/site-packages/datahub/__init__.py',\n"
           " 'py_version': '3.10.9 (main, Dec  8 2022, 01:35:40) [GCC 10.2.1 20210110]',\n"
           " 'py_exec_path': '/tmp/datahub/ingest/venv-powerbi-0.9.6/bin/python3',\n"
           " 'os_details': 'Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.31',\n"
           " 'mem_info': '68.89 MB'}\n"
           'Source (powerbi) report:\n'
           "{'events_produced': '0',\n"
           " 'events_produced_per_sec': '0',\n"
           " 'event_ids': [],\n"
           " 'warnings': {},\n"
           " 'failures': {},\n"
           " 'dashboards_scanned': '0',\n"
           " 'charts_scanned': '0',\n"
           " 'filtered_dashboards': [],\n"
           " 'filtered_charts': [],\n"
           " 'start_time': '2023-01-24 14:28:17.169216 (1.19 seconds ago).',\n"
           " 'running_time': '1.19 seconds'}\n"
           'Sink (datahub-rest) report:\n'
           "{'total_records_written': '0',\n"
           " 'records_written_per_second': '0',\n"
           " 'warnings': [],\n"
           " 'failures': [],\n"
           " 'start_time': '2023-01-24 14:28:17.069492 (1.29 seconds ago).',\n"
           " 'current_time': '2023-01-24 14:28:18.360552 (now).',\n"
           " 'total_duration_in_seconds': '1.29',\n"
           " 'gms_version': 'v0.9.6',\n"
           " 'pending_requests': '0'}\n"
           '\n'
           ' Pipeline finished successfully; produced 0 events in 1.19 seconds.\n',
           "2023-01-24 14:28:20.932903 [exec_id=d2313617-911c-4a97-8d25-0679f414c08e] INFO: Successfully executed 'datahub ingest'"],
 'structured_report': '{"cli": {"cli_version": "0.9.6", "cli_entry_location": '
                      '"/tmp/datahub/ingest/venv-powerbi-0.9.6/lib/python3.10/site-packages/datahub/__init__.py", "py_version": "3.10.9 (main, Dec  '
                      '8 2022, 01:35:40) [GCC 10.2.1 20210110]", "py_exec_path": "/tmp/datahub/ingest/venv-powerbi-0.9.6/bin/python3", "os_details": '
                      '"Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.31", "mem_info": "68.89 MB"}, "source": {"type": "powerbi", "report": '
                      '{"events_produced": "0", "events_produced_per_sec": "0", "event_ids": [], "warnings": {}, "failures": {}, '
                      '"dashboards_scanned": "0", "charts_scanned": "0", "filtered_dashboards": [], "filtered_charts": [], "start_time": "2023-01-24 '
                      '14:28:17.169216 (now).", "running_time": "0.82 seconds"}}, "sink": {"type": "datahub-rest", "report": '
                      '{"total_records_written": "0", "records_written_per_second": "0", "warnings": [], "failures": [], "start_time": "2023-01-24 '
                      '14:28:17.069492 (now).", "current_time": "2023-01-24 14:28:17.989186 (now).", "total_duration_in_seconds": "0.92", '
                      '"gms_version": "v0.9.6", "pending_requests": "0"}}}'}
Execution finished successfully!
The thing is, I'm sure there are a lot of reports, dashboards, datasets, etc. Could you help me to guess what I am doing wrong? Thank you in advance!!!!!!!!!!yay bear
b
Hi ! I have exactly the same issue. I'm interested in the solution if anyone has it thanks!
After several tests, I manage to find that the service principal needs to have the admin right in the workspace that you want to scan. Once I have added it with this role, I've got a successfull result scan for the dashboard. But it is not yet fully resolved because I'm still getting an error when I want to scan the dataset and report... Hope that's help a little bit.
s
Hi all, did you managed to solve this problem? I have exactly the same issue :(
c
Hi All, does any one solved this problem, we are also facing the same issue. Can someone help me, please? Pipeline finished successfully; produced 0 events in 11.97 seconds.
plus1 3