Hi, I'm having issues writing data to BigQuery fr...
# replication-troubleshooting
b
Hi, I'm having issues writing data to BigQuery from Facebook Marketing Source, when using the GCS Staging Method. I have assigned the following permissions to the service account. See Screenshot. • BigQuery Data Editor (Was not sure with provided documentation if needed when using GCS found a hint on Slack) • BigQuery Job User (Was not sure with provided documentation if needed when using GCS found a hint on Slack) • BigQuery User (Was not sure with provided documentation if needed when using GCS found a hint on Slack) • Storage Object Admin I also generated an HMAC Key + Secret for the Bucket as described here: https://airbytehq.github.io/integrations/destinations/bigquery/#recommended-using-a-google-cloud-storage-bucket and here: https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create This confused me though:
• Make sure the HMAC key is created for the BigQuery service account, and the service account has permission to access the GCS bucket and path.
Additionally added the optional Service Account Key JSON (Not sure what that would do) But I'm still getting permission errors.
Copy code
2022-11-27 07:43:14 - Additional Failure Information: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID: null
Strangely it is talking about Amazon S3. See in the screenshot bellow. The Connection test when saving the connection passes though. When clicking retest connection I don't see an error nor a success message though.
🦆 1
✅ 1
I had a look through the logs on GCP: There I see the following errors:
Already Exists: Dataset mydataset-s2s-bigquery:_airbyte_airbyte
also I see the following:
Table mydataset:airbyte._airbyte_raw_ads_insights was not found in
When I look in BigQuery I see that a new dataset Airbyte was created with no data inside.
Just rechecked the HMAC key to make sure this is not the issue.
Found the issue. The gcs bucket path ended with a slash. Was
data_sync/
instead of
data_sync
The error message was confusing though.
Could be improved by adding this to the info and improving the connection test to check this.
y
I’ve been having the same issue @Benedikt Buchert Thank you for your message. However despite your solution, still blocking 😢