@kapa.ai I consistently get this error on the Klaviyo source, for large data, with a BigQuery destination

Warning from destination: 500 Internal Server Error PUT... An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem

The error you're seeing is a BigQuery 500 Internal Server Error during the upload/load phase of your sync. Based on the knowledge sources, here's what's known about this issue: What's Happening This is a transient error from BigQuery's side during the data load phase. It has been observed in Airbyte's Klaviyo → BigQuery pipeline specifically, and is described as: > "An internal error occurred and the request could not be completed. This is usually caused by a transient issue. Retrying the job with back-off as described in the BigQuery SLA should solve the problem." [BigQuery 500 error] Key Concern: Runaway Retries A significant known issue with this scenario is that Airbyte may automatically retry the sync many times, which for large data volumes can cause massive unexpected costs. A documented real-world case involved: • Klaviyo → BigQuery sync extracting >100 GB successfully, but failing during the load phase • The system automatically retried 23 times, transferring the same 100+ GB repeatedly • No built-in mechanism to stop this behavior [Feature Request] What You Can Do 1. Check if it resolves on its own: Since this is flagged as a transient BigQuery issue, it may self-resolve after a few retries. Monitor your sync closely. 2. Monitor for runaway retries: If you have large data volumes, watch your credit/cost consumption carefully, as retries can be excessive and there is currently no user-configurable retry limit. 3. Keep Airbyte and connectors up to date: Updating to the latest versions of Airbyte and your connectors has resolved similar issues for other users. [Forum discussion] 4. If the error persists: File a GitHub issue with your logs at github.com/airbytehq/airbyte/issues, as the Airbyte team tracks these for fixes. [Recharge docs] Longer-Term There is an open feature request to add configurable retry limits and cost circuit breakers at the connection level to prevent runaway retries on large syncs. If this impacts you significantly, you may want to upvote or comment on that discussion. [Feature Request] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai The error is definitely not actually transient as the connection has been failing for weeks. How would Batched Standard Inserts vs. GCS Staging be at play

Since the error has persisted for weeks, it's worth understanding how the two loading methods differ and what issues are associated with each. Batched Standard Inserts vs. GCS Staging Batched Standard Inserts uses the BigQuery driver to convert large INSERT statements into file uploads automatically. This method is generally not recommended for large syncs — Airbyte support has explicitly stated: > "This method is only recommended for small tests; for large syncs you should use GCS Staging." [MySQL→BigQuery forum] GCS Staging writes large batches of records to a file, uploads the file to GCS, then uses

COPY INTO

to load data into BigQuery. [Terraform docs] This is the recommended approach for large data volumes. Known Issues with GCS Staging on Large Syncs However, GCS Staging is not without its own problems for large, long-running syncs: • Authentication token expiry: The "invalid euc header" (End-User Credentials) error during resumable uploads suggests authentication tokens may expire during long-running operations, causing retries that restart from the same position without committing checkpoints. [GitHub issue] • Sync stuck in retry loop: Checkpoints may flush internally but not persist to the connection state, meaning the sync keeps retrying from the same cursor position. [GitHub issue] • Performance bottleneck: Only ~2–4 CPU cores are utilized during GCS staging loads, making it slow for large datasets. [GitHub issue] Recommendation If you're currently on Batched Standard Inserts, switching to GCS Staging is the right first step for large Klaviyo data. However, be aware that GCS Staging has its own open issues with long-running syncs, particularly around token expiry and checkpoint persistence. The knowledge sources do not contain a definitive fix for the persistent 500 error in your specific scenario beyond these known issues. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.