Hi guys, i tried to ingest biguqery-usage for 2 pr...
# ingestionf
few-air-56117
02/03/2022, 7:13 AMHi guys, i tried to ingest biguqery-usage for 2 project, its started , but after 2-3 minutes i get this error
Copy code
Quota exceeded for quota metric 'Read requests' and limit 'Read requests per minute' of service '<http://logging.googleapis.com|logging.googleapis.com>' for consumer 'project_number:491986273194'. [{'@type': '<http://type.googleapis.com/google.rpc.ErrorInfo|type.googleapis.com/google.rpc.ErrorInfo>', 'reason': 'RATE_LIMIT_EXCEEDED', 'domain': '<http://googleapis.com|googleapis.com>', 'metadata': {'consumer': 'projects/491986273194', 'quota_metric': '<http://logging.googleapis.com/read_requests|logging.googleapis.com/read_requests>', 'quota_limit': 'ReadRequestsPerMinutePerProject', 'service': '<http://logging.googleapis.com|logging.googleapis.com>'}}] Copy code
source:
type: bigquery-usage
config:
# Coordinates
projects:
- <project1>
- <project2>
max_query_duration: 5
sink:
type: "datahub-rest"
config:
server: <ip> Copy code
linkedin/datahub-ingestion:v0.8.24 Copy code
args: ["ingest", "-c", "file"]✅ 1
few-air-56117
02/03/2022, 7:13 AMAlso i think if i run on local, via cli, it will work,but i will test it.
d
dazzling-judge-80093
02/03/2022, 7:16 AMAre you saying that it works if you run locally but you get rate limited if you run from cronjob?
Do you use the same service account?
Do you ingest the same data?
f
few-air-56117
02/03/2022, 7:17 AMi remember that i run it via cli on my pc an it works, but i wil try to do a test also to me sure.
few-air-56117
02/03/2022, 7:17 AMYea , i use a service account on k8s and local (the same service account)
few-air-56117
02/03/2022, 7:22 AMi belive that is this api
few-air-56117
02/03/2022, 7:24 AMi also start a local run, i will come back after it finish
few-air-56117
02/03/2022, 7:28 AMyep, on local it work
Copy code
{'records_written': 2356,
'warnings': [],
'failures': [],
'downstream_start_time': datetime.datetime(2022, 2, 3, 9, 22, 22, 506673),
'downstream_end_time': datetime.datetime(2022, 2, 3, 9, 28, 35, 830540),
'downstream_total_latency_in_seconds': 373.323867}few-air-56117
02/03/2022, 7:29 AMstrange 😂
d
dazzling-judge-80093
02/03/2022, 7:35 AMwow
dazzling-judge-80093
02/03/2022, 7:35 AMIs this the same ingestion recipe and the same service account?
f
few-air-56117
02/03/2022, 7:37 AMyep
few-air-56117
02/03/2022, 7:39 AMi think the problem is on the project where i have the datahub, probably google have some limit and i need to increse it
d
dazzling-judge-80093
02/03/2022, 7:41 AMit can be, it seems like you can edit quotas -> https://cloud.google.com/logging/quotas
f
few-air-56117
02/03/2022, 7:42 AMyea, right now i have only 60
few-air-56117
02/03/2022, 7:42 AMi will edit it , but i need to w8 google to approve it
b
bland-lighter-26751
12/12/2022, 6:19 PMDid updating the quota fix this? Also how much did you increase it?
bland-lighter-26751
12/12/2022, 6:36 PMNvm, never realized I had my recipe looking way back in the past. Set it to check back only a week and it's fine
3 Views