Hi all, how i tried to ingest data from biguery an...
# ingestionf
few-air-56117
12/17/2021, 7:57 AMHi all, how i tried to ingest data from biguery and make lineage automaticaly, this is the config
Copy code
source:
type: bigquery
config:
project_id: <project_id>
include_table_lineage: True
sink:
type: "datahub-rest"
config:
server: "<http://localhost:8080>"d
dazzling-judge-80093
12/17/2021, 8:02 AMWhat do you see in the ingest output on console?
Is there any error?
f
few-air-56117
12/17/2021, 8:03 AM Copy code
{'records_written': 547,
'warnings': [],
'failures': [],
'downstream_start_time': datetime.datetime(2021, 12, 17, 9, 56, 27, 551337),
'downstream_end_time': datetime.datetime(2021, 12, 17, 10, 3, 3, 533118),
'downstream_total_latency_in_seconds': 395.981781}
Pipeline finished successfullyfew-air-56117
12/17/2021, 8:03 AMwll, i just runnit again and its work now
few-air-56117
12/17/2021, 8:05 AMHmm, maybe because of start time and end time?
few-air-56117
12/17/2021, 8:06 AMI run the view today and i think is because of this
few-air-56117
12/17/2021, 8:29 AMBut its looks like the downstream lineage dosent work. I create a table base an a view an re run the ingestion and dosent apear in lineage
f
fierce-alligator-27212
02/22/2022, 3:52 PMDid you fix it? @few-air-56117 We can’t get the lineage button to activate
f
few-air-56117
02/22/2022, 3:58 PMyep, the start time and end time default values are today.you can change the start time
Copy code
start_time: 2022-02-11 12:00:00.065028Z
end_time: 2022-02-11 17:00:00.065028Zfew-air-56117
02/22/2022, 3:58 PMdatahub use logs to check the queries, thats why you need start time and end time
f
fierce-alligator-27212
02/22/2022, 4:04 PMWe did update the time parameters but doesn’t look like it was able to get any entries back. It says
0 entires Copy code
[2022-02-22 10:38:24,592] INFO {datahub.cli.ingest_cli:86} - Starting metadata ingestion
[2022-02-22 10:38:24,592] INFO {datahub.ingestion.source.sql.bigquery:320} - Populating lineage info via GCP audit logs
[2022-02-22 10:38:25,997] INFO {datahub.ingestion.source.sql.bigquery:381} - Start loading log entries from BigQuery
[2022-02-22 11:00:36,725] INFO {datahub.ingestion.source.sql.bigquery:520} - Creating lineage map: total number of entries=0, number skipped=0.
[2022-02-22 11:00:36,726] INFO {datahub.ingestion.source.sql.bigquery:316} - Built lineage map containing 0 entries.f
few-air-56117
02/22/2022, 4:21 PMcan you share de config yaml?
few-air-56117
02/22/2022, 4:23 PMtry
Copy code
start_time: 2022-01-01 00:00:00Z
end_time: 2022-02-21 00:00:00Zfew-air-56117
02/22/2022, 4:23 PMyour start time and end time was equal
few-air-56117
02/22/2022, 4:23 PMyour example
Copy code
start_time: 2022-02-20 00:00:00Z
end_time: 2022-02-21 00:00:00Zf
fierce-alligator-27212
02/22/2022, 4:39 PMThank you @few-air-56117 . I added the config file here. https://datahubspace.slack.com/archives/CUMUWQU66/p1645546702043999
10 Views