Hi all, did anyone had speed problem when they t...
# ingestionf
few-air-56117
12/20/2021, 2:37 PMHi all, did anyone had speed problem when they tried to ingest data from bigquery with a 1/2 years gaps between start date and end date?
d
dazzling-judge-80093
12/20/2021, 2:39 PMWhat do you mean in speed problem?
Does profiling/ingestion enabled for you?
Is this
bigquery-usagebigqueryf
few-air-56117
12/20/2021, 2:46 PMNo problem, also i am beginner on datahub
few-air-56117
12/20/2021, 2:47 PMThe bigquery source
few-air-56117
12/20/2021, 2:47 PMthis is the recepie
Copy code
source:
type: bigquery
config:
project_id: <project_id>
# options: # options is same as above
# See <https://github.com/mxmzdlv/pybigquery#authentication> for details
# include_views: True # whether to include views, defaults to Tr
include_table_lineage: True
max_query_duration: 9
start_time: 2018-11-16 00:00:00.065028Z
end_time: 2021-12-17 23:59:00.065028Z
sink:
type: "datahub-rest"
config:
server: "<http://localhost:8080>"few-air-56117
12/20/2021, 2:47 PMandd its up for 2 hour and nothing. Maybe because its a massive amount of logs
d
dazzling-judge-80093
12/20/2021, 5:03 PMAhh, I see then it seems like the table_lineage does something not very effectively
f
few-air-56117
12/20/2021, 5:29 PMYea 😅
d
dazzling-judge-80093
12/20/2021, 7:06 PMI will try to check why it can be so slow but in the meantime can you ingest in smaller chunks?
f
few-air-56117
12/21/2021, 6:27 AMYep, for one month its oky
few-air-56117
12/21/2021, 6:32 AMIts posible to automate the ingestion command ? With Python/js?
few-air-56117
12/21/2021, 7:08 AMfor 1 year looks ok but if i tried 2-3 years its just looping
few-air-56117
12/21/2021, 7:20 AMor if i deploy it on google cloud, can i schedule the ingestion?