Hi, we are trying to enable lineage info for BigQuery. We can see all the datasets/tables in the UI but not the lineage info. Based on the logs, it seems that it wasn’t able to get any entries back. Wonder if anyone ran into this issue and potential causes. Thanks.

[2022-02-22 10:38:24,592] INFO     {datahub.cli.ingest_cli:86} - Starting metadata ingestion
[2022-02-22 10:38:24,592] INFO     {datahub.ingestion.source.sql.bigquery:320} - Populating lineage info via GCP audit logs
[2022-02-22 10:38:25,997] INFO     {datahub.ingestion.source.sql.bigquery:381} - Start loading log entries from BigQuery
[2022-02-22 11:00:36,725] INFO     {datahub.ingestion.source.sql.bigquery:520} - Creating lineage map: total number of entries=0, number skipped=0.
[2022-02-22 11:00:36,726] INFO     {datahub.ingestion.source.sql.bigquery:316} - Built lineage map containing 0 entries.

config:

source:
  type: bigquery
  config:
    project_id: <GCP Project ID>
    env: prod
    include_table_lineage: True
    start_time: 2022-02-20 00:00:00Z
    end_time: 2022-02-21 00:00:00Z

sink:
  type: "datahub-rest"
  config:
    server: "<http://localhost:8080>"

no help, but fwiw we also haven't had any luck getting lineage from BQ logs

do you have any table/view created in start time, end time range?

Ooo, that might be the part I am missing. I will let you know! 🙂

you can made 1 full load, from 2021 01 01 , and after that you can remove the start date and end date

Oh maybe we should try this as well 👀

this is how i do. Google logs are for 1~2 years, so i do a manual igestion using the cli

datahub ingest -c <file.yaml>

with start date 2021-01-01 And, after that , i schedule the igestion via ui without start date/end date

Hi everyone (and summon @fierce-alligator-27212, @few-air-56117), Just follow-up on this thread, as I see this is more relavant for my case. I want to enable BigQuery lineage in my DataHub cluster, spent couple of days to understand why it didn't show up. Need a help! Here is the log creation,

$ datahub ingest -c awesome-analytics.yml -n
[2022-03-31 11:48:48,997] INFO     {datahub.cli.ingest_cli:75} - DataHub CLI version: 0.8.31.2
[2022-03-31 11:48:49,101] INFO     {datahub.ingestion.sink.datahub_rest:60} - Setting gms config
[2022-03-31 11:48:58,870] INFO     {datahub.cli.ingest_cli:91} - Starting metadata ingestion
[2022-03-31 11:48:58,870] INFO     {datahub.ingestion.source.sql.bigquery:276} - Populating lineage info via GCP audit logs
[2022-03-31 11:48:58,918] INFO     {datahub.ingestion.source.sql.bigquery:368} - Start loading log entries from BigQuery start_time=2021-12-31T23:45:00Z and end_time=2022-12-31T00:15:00Z




[2022-03-31 11:59:09,824] INFO     {datahub.ingestion.source.sql.bigquery:379} - Finished loading 12575 log entries from BigQuery so far
[2022-03-31 11:59:09,824] INFO     {datahub.ingestion.source.sql.bigquery:461} - Parsing BigQuery log entries: number of log entries successfully parsed=12575
[2022-03-31 11:59:09,824] INFO     {datahub.ingestion.source.sql.bigquery:512} - Creating lineage map: total number of entries=12575, number skipped=90.
[2022-03-31 11:59:09,824] INFO     {datahub.ingestion.source.sql.bigquery:269} - Built lineage map containing 9492 entries.
[2022-03-31 11:59:10,566] INFO     {datahub.cli.ingest_cli:93} - Finished metadata ingestion

Source (bigquery) report:
{'workunits_produced': 3,
 'workunit_ids': ['container-info-awesome-analytics-urn:li:container:f175a2a3942345da2bcd53c03b6f7033',
                  'container-platforminstance-awesome-analytics-urn:li:container:f175a2a3942345da2bcd53c03b6f7033',
                  'container-subtypes-awesome-analytics-urn:li:container:f175a2a3942345da2bcd53c03b6f7033'],
 'warnings': {},
 'failures': {},
 'cli_version': '0.8.31.2',
 'cli_entry_location': '/home/mfx/datahub-dev/venv/lib/python3.9/site-packages/datahub/__init__.py',
 'py_version': '3.9.2 (default, Feb 28 2021, 17:03:44) \n[GCC 10.2.1 20210110]',
 'py_exec_path': '/home/mfx/datahub-dev/venv/bin/python3',
 'os_details': 'Linux-5.10.90+-x86_64-with-glibc2.31',
 'tables_scanned': 0,
 'views_scanned': 0,
 'entities_profiled': 0,
 'filtered': [],
 'soft_deleted_stale_entities': [],
 'query_combiner': None,
 'num_total_log_entries': 12575,
 'num_parsed_log_entires': 12575,
 'num_total_audit_entries': None,
 'num_parsed_audit_entires': None,
 'lineage_metadata_entries': 9492,
 'use_v2_audit_metadata': False,
 'use_exported_bigquery_audit_metadata': False,
 'start_time': datetime.datetime(2022, 1, 1, 0, 0, tzinfo=datetime.timezone.utc),
 'end_time': datetime.datetime(2022, 12, 31, 0, 0, tzinfo=datetime.timezone.utc),
 'log_entry_start_time': '2021-12-31T23:45:00Z',
 'log_entry_end_time': '2022-12-31T00:15:00Z',
 'audit_start_time': None,
 'audit_end_time': None}
Sink (datahub-rest) report:
{'records_written': 0,
 'warnings': [],
 'failures': [],
 'downstream_start_time': None,
 'downstream_end_time': None,
 'downstream_total_latency_in_seconds': None,
 'gms_version': 'v0.8.31'}

Is there anything I missed the most? Thank you

Hi, only bigquery has this config, and other sources don't. what should I do, such as Oracal, mssql🙃🙃