Is this your first time deploying Airbyte?: No OS ...
# replication-troubleshooting
n
Is this your first time deploying Airbyte?: No OS Version / Instance: linux Memory / Disk: 4GB / 30 GB (per node) - up to 5 nodes Deployment: Kubernetes (Stable with resource limits) Airbyte Version: 0.40.18 Source name/version: Stripe (Latest) (Connection Test Successful) Destination name/version: Big Query (Latest) (Connection Test Successful) Step: During First Sync up. I've tried this many times. all the time Source connection check completes with success status. Destination check does not complete and start showing errors like in the log file below. (this log file does not contains source and destination checks but the error is same). Please help me here.
n
Hey, Nikhil! I'm looking into this and have found a few different instances of people running into a similar issue with the BigQuery destination. I don't see any conclusive solutions yet, but here's what I've found so far: https://discuss.airbyte.io/t/kubernetes-bigquery-sync-failed/1222 https://discuss.airbyte.io/t/unable-to-sync-data-from-mssql-to-big-query/2986 https://discuss.airbyte.io/t/destination-bigquery-normalization-error-with-cdc/2839 https://github.com/airbytehq/airbyte/issues/17100 So I have a few questions - 1. What sync mode are you using? 2. Do you see any tables being created in BigQuery? 3. For good measure, could you update everything?
n
Hey @Nataly Merezhuk (Airbyte) thank you for responding. I've update on this. I think Big query connection might not be an issue here. Here is the whole scenario. I've already set up all the air byte connections on kubernetes. but I wanted to do logging in GCS Buckets. (using this link https://docs.airbyte.com/deploying-airbyte/on-kubernetes#configure-the-custom-gcs-log-location). So I've created a new kubernetes cluster and set up GCS logging Errors started after this set up. I wasn't sure what went wrong in while sync up, I only see destination check was not completed (according to log file), so that's what I've mentioned in my above message. But when I removed GCS Logging and do sync up, all started worked perfectly. No error what so ever. I am still looking to implement GCS logging with my Kubernetes set up. Here are few thing to know, 1. GCS logging was working and put logs files in GCS buckets. (FYI: All logs I was seeing was about server api logs) 2. Big Query connection was set up with GCS stagging. I've tried same bucket and different bucket than logging, gave me an error as shown in log file above. Let me know, if you need any other information in this.
n
@Nikhil Patel could you please share your kubernetes config file for the GCS Logging? Also please take a look at this issues, as they are directly related and may help: https://github.com/airbytehq/airbyte/issues/19475 https://github.com/airbytehq/airbyte/issues/17624