Hi guys,
I'm testing the bigquery ingestion, and f...
# troubleshootn
numerous-application-54063
02/18/2022, 3:19 PMHi guys,
I'm testing the bigquery ingestion, and for metadata works fine but i don't get any lineage or usage info.
the cli (version 0.8.26.3) says parsing 0 rows from logs. So i tried to execute the log filter on gcp console and when i filter with this condition:
protoPayload.serviceName="<http://bigquery.googleapis.com|bigquery.googleapis.com>"protoPayload.methodName="jobservice.jobcompleted"methodName: "google.cloud.bigquery.v2.JobService.InsertJob"d
dazzling-judge-80093
02/18/2022, 3:22 PMLet me look after this…
n
numerous-application-54063
02/18/2022, 3:25 PMthanks!
d
dazzling-judge-80093
02/18/2022, 3:51 PMI need to do some investigation what is the difference between the two and to understand if it can happen one returns with result while the other not.
n
numerous-application-54063
02/18/2022, 4:05 PMok, let me know if i can give you further info
d
dazzling-judge-80093
02/21/2022, 2:44 PMhey,
Could you check me please if these returns queries?
Copy code
protoPayload.serviceName="<http://bigquery.googleapis.com|bigquery.googleapis.com>" AND protoPayload.methodName="google.cloud.bigquery.v2.JobService.Query" Copy code
protoPayload.serviceName="<http://bigquery.googleapis.com|bigquery.googleapis.com>" AND protoPayload.methodName="jobservice.jobcompleted"n
numerous-application-54063
02/21/2022, 3:19 PMHey, i actually get no results for both
d
dazzling-judge-80093
02/21/2022, 4:37 PMAre you sure you ran some query in bigquery before?
I would like to understand what is happening.
n
numerous-application-54063
02/21/2022, 4:42 PMyes we have several airflow dags running.
but i can see only logs with methodName = google.cloud.bigquery.v2.JobService.InsertJob.
i agree that this sounds strange
d
dazzling-judge-80093
02/21/2022, 5:40 PMthanks, I try to research further and will get back to you
dazzling-judge-80093
02/21/2022, 10:08 PMHey,
I rewrote the query by the audit v2 format.
Can you check this as well ->
Copy code
resource.type=("bigquery_project") AND protoPayload.methodName=("google.cloud.bigquery.v2.JobService.Query" OR "google.cloud.bigquery.v2.JobService.InsertJob") AND protoPayload.metadata.jobChange.job.jobStatus.jobState="DONE" AND NOT protoPayload.metadata.jobChange.job.jobStatus.errorResult:*n
numerous-application-54063
02/22/2022, 10:20 AMHey, yes with this one i'm getting bigquery related logs!
d
dazzling-judge-80093
02/22/2022, 12:07 PMInteresting but great to hear 🙂
For me both gives back the same results but based on the documentation v2 is the recommended way to use.
n
numerous-application-54063
02/22/2022, 12:08 PMyes maybe it's due to some particular configuration in our environment, not sure about it. Thanks for taking the time!
d
dazzling-judge-80093
02/22/2022, 1:16 PMOther users are facing this issue as well and this seems to be promising to solve it.
🚀 1
dazzling-judge-80093
03/08/2022, 10:58 AM@numerous-application-54063 We cut a new python release yesterday and there there is an option to enable the v2 api.
I wonder if you could give it a go and test if this helps.
To enable it you should set in your bigquery source recipe:
use_v2_audit_metadata: Truen
numerous-application-54063
03/08/2022, 4:56 PMi tested it and it was still not working, but now can also be a permission issue on my side, since the query seems to work in the cloud logging ui.
i will try to have a more detailed look at this and get back to you 🙂
d
dazzling-judge-80093
03/08/2022, 5:42 PMThanks