You need to set `pipeline_name` -> <https://dat...
# ingestiond
dazzling-judge-80093
02/25/2022, 11:13 AMYou need to set
pipeline_namef
few-air-56117
02/25/2022, 11:15 AMoh, i see, thx Tamas , You helped me many times :d
d
dazzling-judge-80093
02/25/2022, 11:18 AMI’m happy to help 🙂
f
few-air-56117
02/25/2022, 11:37 AMi run the ingestion (after i drop a view) but the view is still in datahub
recepie
Copy code
source:
type: bigquery
config:
project_id: project1
credential:
include_table_lineage: true
stateful_ingestion:
enabled: true
sink:
type: datahub-rest
config:
server: '<http://localhost:8080>'
pipeline_name: "test_bq" Copy code
'tables_scanned': 384,
'views_scanned': 197,
'entities_profiled': 0,
'filtered': [],
'soft_deleted_stale_entities': [],
'query_combiner': None}
Sink (datahub-rest) report:
{'records_written': 581,
'warnings': [],
'failures': [],
'downstream_start_time': datetime.datetime(2022, 2, 25, 13, 20, 35, 953359),
'downstream_end_time': datetime.datetime(2022, 2, 25, 13, 26, 11, 524126),
'downstream_total_latency_in_seconds': 335.570767} Copy code
BigQuery Admin
BigQuery Metadata Viewer
Logging Admind
dazzling-judge-80093
02/25/2022, 1:58 PMIn the logs at the beginning there should be some logline about how many audit log got from Bq and how many were used for lineage building.
Please, can you check it?
We saw some users who failed to have lineage and I’m chasing this and I have a possible fix which will be available soon. -> I do hope it will fix but as I couldn’t reproduce this issue on our side.
f
few-air-56117
02/25/2022, 2:22 PMoh, this warning
few-air-56117
02/25/2022, 2:22 PM{datahub.ingestion.source.sql.bigquery:224} - Built lineage map containing 0 entries.
/Users/dragos.c-ami/work/myvirtualenv/lib/python3.8/site-packages/google/cloud/bigquery/client.py513 UserWarning: Cannot create BigQuery Storage client, the dependency google-cloud-bigquery-storage is not installed.
few-air-56117
02/25/2022, 2:29 PMhmm, i think i need some extra right. I run for another project i have Built lineage map containing 3856 entries.
few-air-56117
02/25/2022, 2:43 PM2 project, same right, for one i got Built lineage map containing 3856 entries. and for the second one i got Built lineage map containing 0 entries.
few-air-56117
02/28/2022, 9:09 AMhi @dazzling-judge-80093, i think the view is not deleted because i got this errror
few-air-56117
02/28/2022, 9:09 AM Copy code
Querying for the latest ingestion checkpoint for pipelineName:'test_bq', platformInstanceId:'bigquery_no_host_port_no_database', job_name:'common_ingest_from_sql_source'
[2022-02-28 11:08:30,356] INFO {datahub.ingestion.run.pipeline:79} - sink wrote workunit am-dwh-t1.tv.P_VD_TV
[2022-02-28 11:08:30,650] INFO {datahub.ingestion.source.state_provider.datahub_ingestion_state_provider:89} - The last committed ingestion checkpoint for pipelineName:'test_bq', platformInstanceId:'bigquery_no_host_port_no_database', job_name:'common_ingest_from_sql_source' found with start_time: 2022-02-28 07:01:07.713000+00:00 and a bucket duration of None.
[2022-02-28 11:08:30,653] INFO {datahub.ingestion.source.state.checkpoint:118} - Successfully constructed last checkpoint state for job common_ingest_from_sql_source
[2022-02-28 11:08:30,656] INFO {datahub.ingestion.source.state_provider.datahub_ingestion_state_provider:109} - Committing ingestion checkpoint for pipeline:'test_bq',instance:'bigquery_no_host_port_no_database', job:'common_ingest_from_sql_source'
[2022-02-28 11:08:30,821] INFO {datahub.ingestion.source.state_provider.datahub_ingestion_state_provider:130} - Committed ingestion checkpoint for pipeline:'test_bq',instance:'bigquery_no_host_port_no_database', job:'common_ingest_from_sql_source'
[2022-02-28 11:08:30,821] INFO {datahub.cli.ingest_cli:83} - Finished metadata ingestiond
dazzling-judge-80093
02/28/2022, 9:13 AMInterestingly here I can’t see even if it tried to get the audit logs.
Is this log partial or should contain everything?
f
few-air-56117
02/28/2022, 9:16 AMits everything
d
dazzling-judge-80093
02/28/2022, 9:18 AMDo you use two different ingestion recipe of one for both project?
f
few-air-56117
02/28/2022, 9:21 AMi add
Copy code
start_time: 2022-02-28 00:00:00.065028Z
end_time: 2022-02-28 23:59:00.065028Zd
dazzling-judge-80093
02/28/2022, 9:28 AMDoes the lineage work if you split into two part?
5 Views