Hi @kapa.ai. I recently upgraded Airbyte (self hosted), from 1.5.1 > 2.0.1 (helm chart versions 1.5.1 > 1.9.2), and now I'm seeing lots of syncs fail with 'transient errors':

Warning from airbyte_platform: An internal transient Airbyte error has occurred. The sync should work fine on the next retry.

How can I debug/fix this? I have tried disabling and then enabling all connections and also a rolling restart of all deployments

Based on the knowledge sources, there's a relevant case that closely matches your situation. After upgrading from v1.1 to v1.8, users saw similar transient errors:

An internal transient Airbyte error has occurred. The sync should work fine on the next retry. Setting attempt to FAILED because the workflow for this connection was restarted, and existing job state was cleaned.

The root cause in that case was running Temporal with multiple replicas (for HA). The fix was to drop Temporal's replica count to 1. This significantly reduced load on the metadata DB (both CPU usage and number of active connections) and resolved the failing syncs. [High availability Temporal issue] Recommended debugging steps: 1. Check your Temporal replica count — If you're running more than 1 replica of the Temporal server, try scaling it down to 1 and see if the transient errors stop. 2. Check for dataplane group misconfiguration — After upgrades, there have been cases where multiple

dataplane_group

entries exist in the database, causing jobs to be dispatched to one dataplane while the workload launcher listens to another. You can verify this by querying your Airbyte database:

SELECT * FROM public.dataplane_group;
   SELECT id, organization_id, dataplane_group_id FROM public.workspace;

If there are duplicate entries, the workaround is to update the correct

dataplane_group_id

in the workspace table. [airbyte services stuck issue] 3. Check bootloader logs — Ensure the database migration ran successfully after the upgrade:

kubectl logs -n <namespace> <bootloader-pod-name>

[Syncs don't start after upgrade] 4. Verify database connectivity — Confirm the new deployment is pointing to the correct database with your existing data:

kubectl exec -n <namespace> -it <airbyte-db-pod> -- psql -U airbyte -d db-airbyte -c "SELECT id, name FROM workspace LIMIT 5;"

[Syncs don't start after upgrade] Given your large version jump (1.5.1 → 2.0.1), I'd start with checking the Temporal replica count and the dataplane group configuration, as both have been documented causes of sync failures after major upgrades. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

We do have two replicas of temporal running. Will reducing this to 1 not cause reliability issues? @kapa.ai

Yes, reducing Temporal to 1 replica is the recommended action based on the available evidence, and here's why running 2 replicas is actually causing your reliability issues rather than improving them: Why 2 Temporal replicas is problematic The Airbyte Helm chart exposes a

temporal.replicaCount

option, but it does not include the necessary clustering infrastructure to support true Temporal HA. Specifically, it lacks: • Membership configuration in the server configmap • Additional ports for member discovery Without these, the two replicas compete for shard ownership and conflict with each other — which is exactly what causes the transient errors and increased database load you're seeing. [High availability Temporal issue] A contributor confirmed this directly: > "I believe the two replicas were competing for shard ownership and conflicting with each other, causing all those problems." [High availability Temporal issue] What to do Scale Temporal down to 1 replica in your `values.yaml`:

temporal:
  replicaCount: 1

This is a known open issue with the Airbyte Helm chart, and the Airbyte team has been notified for tracking and prioritization. [High availability Temporal issue] On reliability concerns The knowledge sources do not provide an alternative HA solution for Temporal within the current Airbyte Helm chart. For overall Airbyte HA, the documentation recommends running multiple replicas of

worker

and

server

pods (not Temporal), and using a primary-replica setup for PostgreSQL. [Scaling Airbyte] In short: running 2 Temporal replicas in the current setup is reducing reliability, not improving it. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai is this a new issue though? We have been running 2 temporal replicas for a while without problems. Are there changes in my upgrade path which would have started causing problems?

That's a fair and important question. Based on the knowledge sources, there is direct evidence that this became a problem specifically after an Airbyte version upgrade: > "After upgrading Airbyte from

v1.1

to

v1.8

, we saw the load on the metadata DB increasing significantly... We always used Temporal with two replicas for HA. And in the new version Airbyte was failing a lot of syncs." [High availability Temporal issue] This matches your situation closely — you ran 2 Temporal replicas without issues before, and problems started after upgrading. The knowledge sources confirm this is a pattern others have experienced. However, the knowledge sources do not explicitly explain what changed internally between versions that caused 2 Temporal replicas to become problematic after the upgrade. I cannot speculate on the specific internal changes in the 1.5.1 → 2.0.1 path that triggered this. What the sources do confirm is: • The issue is not new in absolute terms (reported across multiple versions including v1.8 and v2.0.x) • It is triggered or worsened by upgrades, even when 2 replicas worked fine before • Dropping to 1 replica resolved it in documented cases [High availability Temporal issue] Given this evidence, reducing to 1 Temporal replica remains the most well-supported fix for your situation, even if it worked fine before your upgrade. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.