@kapa.ai We have a Kafka to BigQuery connection. Since a few months, it is missing a small portion of events (up to 0.6% events lost). What could be the explanations for Airbyte missing some events? It is important for us to get all events synced.

I'm sorry, but the knowledge sources do not contain enough specific information about Kafka-to-BigQuery event loss in Airbyte to give you a fully confident answer. The available documents touch on related but distinct issues. Here is what I can share based on the knowledge sources: Possible Explanations for Missing Events 1. Kafka Consumer Not Connecting Properly One forum post describes a Kafka source reading 0 messages, where the sysadmin confirmed no consumer was connecting to Kafka at all. This suggests connectivity or permission issues can silently drop events. [Kafka 0 records] 2. Auto Offset Reset Configuration The same post mentions switching Auto Offset Reset between

latest

and

earliest

as a troubleshooting step. If set to

latest

, events produced while the connector is not running could be missed permanently. [Kafka 0 records] 3. Destination BigQuery Version Bugs One forum post notes that certain BigQuery destination versions have known bugs (e.g., BigQuery destination 1.21 was listed as bugged in the changelog). Using an outdated or buggy destination connector version could contribute to data loss. [Shopify to BigQuery] 4. BigQuery Destination Failing to Commit All Records There is an open issue where large syncs to BigQuery complete the load job successfully but fail with a

TransientErrorException

(missing terminal stream status), causing only the first batch to be committed to the final table. This could explain a consistent small percentage of lost records. [BigQuery TransientError] Recommendations • Check your connector versions: Upgrade both the Kafka source and BigQuery destination connectors to the latest versions. • Review Kafka consumer permissions and connectivity: Ensure the Airbyte consumer is consistently connecting and has the right permissions. • Check Auto Offset Reset: Make sure it is set to

earliest

if you cannot afford to miss events. • Review sync logs: Look for

TransientErrorException

or partial commit errors in the BigQuery destination logs. • Open a GitHub issue: Given the specificity of your problem (consistent ~0.6% loss), it may be worth filing a detailed bug report at the Airbyte GitHub repository with logs and connector versions. The knowledge sources do not provide a definitive root cause for your specific scenario, so I'd strongly recommend sharing detailed sync logs with the Airbyte community or support team for further diagnosis. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.