Hi! I set up the ingestion cronjob on kubernetes, ...
# all-things-deploymenta
agreeable-hamburger-38305
11/23/2021, 1:27 AMHi! I set up the ingestion cronjob on kubernetes, but when I run a job from the cronjob this is what I get. Not sure if it’s because I haven’t added the equivalent of the plugin mentioned here https://datahubproject.io/docs/metadata-ingestion/source_docs/bigquery#setup, but I also kinda dont know how
Copy code
[2021-11-23 00:56:07,285] INFO {datahub.cli.ingest_cli:57} - Starting metadata ingestion
/usr/local/lib/python3.8/site-packages/google/cloud/bigquery/client.py:513: UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed.
warnings.warn(
[2021-11-23 01:00:09,074] INFO {datahub.cli.ingest_cli:59} - Finished metadata ingestion
Source (bigquery) report:
{'failures': {}, 'filtered': [], 'tables_scanned': 0, 'views_scanned': 0, 'warnings': {}, 'workunit_ids': [], 'workunits_produced': 0}
Sink (datahub-rest) report:
{'downstream_end_time': None,
'downstream_start_time': None,
'downstream_total_latency_in_seconds': None,
'failures': [],
'records_written': 0,
'warnings': []}
Pipeline finished successfullyagreeable-hamburger-38305
11/23/2021, 1:29 AMI have a similar cronjob set up for bigquery usage, same problem: no error, but it also doesnt do anything
Copy code
[2021-11-22 22:57:12,030] INFO {datahub.cli.ingest_cli:57} - Starting metadata ingestion
[2021-11-22 23:01:11,604] INFO {datahub.cli.ingest_cli:59} - Finished metadata ingestion
Source (bigquery-usage) report:
{'dropped_table': Counter(), 'failures': {}, 'warnings': {}, 'workunit_ids': [], 'workunits_produced': 0}
Sink (datahub-rest) report:
{'downstream_end_time': None,
'downstream_start_time': None,
'downstream_total_latency_in_seconds': None,
'failures': [],
'records_written': 0,
'warnings': []}
Pipeline finished successfullyg
gray-wall-52477
01/06/2022, 11:50 PM@agreeable-hamburger-38305 any luck? I'm doing the same with MySQL and same result