https://datahubproject.io logo
Join Slack
Powered by
Hi, I’m trying to use Datahub to pull in PowerBi r...
# troubleshoot
n
Hi, I’m trying to use Datahub to pull in PowerBi reports but have been having issues. The ingestion is successful but doesn’t produce any reports.
b
Hey @narrow-queen-90189, Is the recipe configured properly
?
n
Yes, let me post it here.
source: type: powerbi config: tenant_id: - workspace_id_pattern: allow: - ‘.*’ env: DEV client_id: - client_secret: - extract_reports: true dataset_type_mapping: PostgreSql: postgres Oracle: oracle
Also our reports are based on Athena, if that makes any difference. @bulky-electrician-72362
Thanks
b
it looks good at first glance. try using file as sink to see what's being produced
n
The file is empty
[ ]
@bulky-electrician-72362 ☝️
a
@gray-shoe-75895 do you have any idea why the above recipe might produce empty list to be ingested?
g
Could you run with debug mode enabled and post the logs here?
n
Sure
@gray-shoe-75895, is it possible to set the debug option from the UI ingestion?
a
Yes - on step 4, under advanced settings, you can tick the debug checkbox
n
@gray-shoe-75895, here is the log.
Copy code
~~~~ Execution Summary - RUN_INGEST ~~~~
Execution finished successfully!
{'exec_id': 'abe3d491-5f24-4542-811e-9905cc992e8c',
 'infos': ['2023-03-05 23:18:35.286937 INFO: Starting execution for task with name=RUN_INGEST',
           "2023-03-05 23:18:39.328802 INFO: Successfully executed 'datahub ingest'",
           '2023-03-05 23:18:39.329058 INFO: Finished execution for task with name=RUN_INGEST'],
 'errors': []}

~~~~ Ingestion Report ~~~~
{
  "cli": {
    "cli_version": "0.10.0",
    "cli_entry_location": "/tmp/datahub/ingest/venv-powerbi-0.10.0/lib/python3.10/site-packages/datahub/__init__.py",
    "py_version": "3.10.9 (main, Jan 23 2023, 22:32:48) [GCC 10.2.1 20210110]",
    "py_exec_path": "/tmp/datahub/ingest/venv-powerbi-0.10.0/bin/python3",
    "os_details": "Linux-5.15.0-1026-aws-x86_64-with-glibc2.31",
    "mem_info": "70.79 MB"
  },
  "source": {
    "type": "powerbi",
    "report": {
      "events_produced": 0,
      "events_produced_per_sec": 0,
      "entities": {},
      "aspects": {},
      "warnings": {},
      "failures": {},
      "dashboards_scanned": 0,
      "charts_scanned": 0,
      "filtered_dashboards": [],
      "filtered_charts": [],
      "start_time": "2023-03-05 23:18:37.270028 (now)",
      "running_time": "0.62 seconds"
    }
  },
  "sink": {
    "type": "file",
    "report": {
      "total_records_written": 0,
      "records_written_per_second": 0,
      "warnings": [],
      "failures": [],
      "start_time": "2023-03-05 23:18:37.148006 (now)",
      "current_time": "2023-03-05 23:18:37.890235 (now)",
      "total_duration_in_seconds": 0.74
    }
  }
}

~~~~ Ingestion Logs ~~~~
Obtaining venv creation lock...
Acquired venv creation lock
venv setup time = 0
This version of datahub supports report-to functionality
datahub --debug ingest run -c /tmp/datahub/ingest/abe3d491-5f24-4542-811e-9905cc992e8c/recipe.yml --report-to /tmp/datahub/ingest/abe3d491-5f24-4542-811e-9905cc992e8c/ingestion_report.json
[2023-03-05 23:18:36,741] DEBUG    {datahub.telemetry.telemetry:219} - Sending init Telemetry
[2023-03-05 23:18:37,002] DEBUG    {datahub.telemetry.telemetry:248} - Sending telemetry for function-call
[2023-03-05 23:18:37,144] INFO     {datahub.cli.ingest_cli:165} - DataHub CLI version: 0.10.0
[2023-03-05 23:18:37,148] DEBUG    {datahub.ingestion.run.pipeline:178} - Sink type file (<class 'datahub.ingestion.sink.file.FileSink'>) configured
[2023-03-05 23:18:37,148] INFO     {datahub.ingestion.run.pipeline:179} - Sink configured successfully. 
[2023-03-05 23:18:37,148] DEBUG    {datahub.ingestion.run.pipeline:253} - Reporter type:file,<class 'datahub.ingestion.reporting.file_reporter.FileReporter'> configured.
/tmp/datahub/ingest/venv-powerbi-0.10.0/lib/python3.10/site-packages/datahub/ingestion/source/powerbi/powerbi.py:867: ConfigurationWarning: env is deprecated and will be removed in a future release. Please use platform_instance instead.
  config = PowerBiDashboardSourceConfig.parse_obj(config_dict)
[2023-03-05 23:18:37,398] INFO     {datahub.ingestion.source.powerbi.proxy:231} - Trying to connect to <https://login.microsoftonline.com/{tenant-id}>
[2023-03-05 23:18:37,398] INFO     {datahub.ingestion.source.powerbi.proxy:349} - Generating PowerBi access token
[2023-03-05 23:18:37,482] INFO     {datahub.ingestion.source.powerbi.proxy:363} - Generated PowerBi access token
[2023-03-05 23:18:37,482] DEBUG    {datahub.ingestion.source.powerbi.proxy:367} - PBIAccessToken=Bearer {token}
[2023-03-05 23:18:37,482] INFO     {datahub.ingestion.source.powerbi.proxy:233} - Able to connect to <https://login.microsoftonline.com/{tenant-id}>
[2023-03-05 23:18:37,482] DEBUG    {datahub.ingestion.source.powerbi.proxy:346} - Returning the cached access token
[2023-03-05 23:18:37,617] INFO     {datahub.ingestion.source.powerbi.proxy:231} - Trying to connect to <https://login.microsoftonline.com/{tenant-id}>
[2023-03-05 23:18:37,617] INFO     {datahub.ingestion.source.powerbi.proxy:349} - Generating PowerBi access token
[2023-03-05 23:18:37,703] INFO     {datahub.ingestion.source.powerbi.proxy:363} - Generated PowerBi access token
[2023-03-05 23:18:37,703] DEBUG    {datahub.ingestion.source.powerbi.proxy:367} - PBIAccessToken=Bearer {token}
[2023-03-05 23:18:37,703] INFO     {datahub.ingestion.source.powerbi.proxy:233} - Able to connect to <https://login.microsoftonline.com/{tenant-id}>
[2023-03-05 23:18:37,703] DEBUG    {datahub.ingestion.run.pipeline:195} - Source type powerbi (<class 'datahub.ingestion.source.powerbi.powerbi.PowerBiDashboardSource'>) configured
[2023-03-05 23:18:37,703] INFO     {datahub.ingestion.run.pipeline:196} - Source configured successfully.
[2023-03-05 23:18:37,705] INFO     {datahub.cli.ingest_cli:120} - Starting metadata ingestion
[2023-03-05 23:18:37,706] INFO     {datahub.ingestion.source.powerbi.powerbi:892} - PowerBi plugin execution is started
[2023-03-05 23:18:37,707] INFO     {datahub.ingestion.source.powerbi.proxy:757} - Request to get groups endpoint URL=<https://api.powerbi.com/v1.0/myorg/groups>
[2023-03-05 23:18:37,707] DEBUG    {datahub.ingestion.source.powerbi.proxy:346} - Returning the cached access token
[2023-03-05 23:18:37,714] DEBUG    {datahub.upgrade.upgrade:125} - server_config:{'models': {}, 'patchCapable': True, 'versions': {'linkedin/datahub': {'version': 'v0.10.0', 'commit': '77d072b5220394b574735b9af14948ef7f9c52ca'}}, 'managedIngestion': {'defaultCliVersion': '0.10.0', 'enabled': True}, 'statefulIngestionCapable': True, 'supportsImpactAnalysis': True, 'telemetry': {'enabledCli': True, 'enabledIngestion': False}, 'datasetUrnNameCasing': False, 'retention': 'true', 'datahub': {'serverType': 'quickstart'}, 'noCode': 'true'}
[2023-03-05 23:18:37,890] INFO     {datahub.ingestion.reporting.file_reporter:52} - Wrote SUCCESS report successfully to <_io.TextIOWrapper name='/tmp/datahub/ingest/abe3d491-5f24-4542-811e-9905cc992e8c/ingestion_report.json' mode='w' encoding='UTF-8'>
[2023-03-05 23:18:37,891] INFO     {datahub.cli.ingest_cli:133} - Finished metadata ingestion
[2023-03-05 23:18:37,891] DEBUG    {datahub.telemetry.telemetry:248} - Sending telemetry for ingest_stats

Cli report:
{'cli_version': '0.10.0',
 'cli_entry_location': '/tmp/datahub/ingest/venv-powerbi-0.10.0/lib/python3.10/site-packages/datahub/__init__.py',
 'py_version': '3.10.9 (main, Jan 23 2023, 22:32:48) [GCC 10.2.1 20210110]',
 'py_exec_path': '/tmp/datahub/ingest/venv-powerbi-0.10.0/bin/python3',
 'os_details': 'Linux-5.15.0-1026-aws-x86_64-with-glibc2.31',
 'mem_info': '70.79 MB'}
Source (powerbi) report:
{'events_produced': 0,
 'events_produced_per_sec': 0,
 'entities': {},
 'aspects': {},
 'warnings': {},
 'failures': {},
 'dashboards_scanned': 0,
 'charts_scanned': 0,
 'filtered_dashboards': [],
 'filtered_charts': [],
 'start_time': '2023-03-05 23:18:37.270028 (now)',
 'running_time': '0.78 seconds'}
Sink (file) report:
{'total_records_written': 0,
 'records_written_per_second': 0,
 'warnings': [],
 'failures': [],
 'start_time': '2023-03-05 23:18:37.148006 (now)',
 'current_time': '2023-03-05 23:18:38.046303 (now)',
 'total_duration_in_seconds': 0.9}

 Pipeline finished successfully; produced 0 events in 0.78 seconds.
[2023-03-05 23:18:38,191] DEBUG    {datahub.upgrade.upgrade:264} - Version stats found: server=ServerVersionStats(current=VersionStats(version=<Version('0.10.0')>, release_date=datetime.datetime(2023, 2, 27, 21, 15, 29, tzinfo=datetime.timezone.utc)), latest=VersionStats(version=<Version('0.10.0')>, release_date=datetime.datetime(2023, 2, 7, 21, 16, 15, tzinfo=datetime.timezone.utc)), current_server_type='quickstart') client=ClientVersionStats(current=VersionStats(version=<Version('0.10.0')>, release_date=datetime.datetime(2023, 2, 8, 15, 40, 32)), latest=VersionStats(version=<Version('0.10.0.4')>, release_date=datetime.datetime(2023, 3, 3, 22, 34, 26)))
[2023-03-05 23:18:38,191] DEBUG    {datahub.telemetry.telemetry:248} - Sending telemetry for function-call
g
This is likely a permissions issue with your access to PowerBI
n
Are there any other permissions that need to be given?
a
Open in Slack
PreviousNext
Powered by