<@U04S8PLJWMT> Warning from airbyte_platform: An i...
# replication-ask-ai
k
@kapa.ai Warning from airbyte_platform: An internal transient Airbyte error has occurred. The sync should work fine on the next retry. Setting attempt to FAILED because the workflow for this connection was restarted, and existing job state was cleaned.
k
This warning indicates a platform-level, transient error in Airbyte’s orchestration layer (Temporal), not a connector bug. From the forum and GitHub issues, this message appears when the Temporal workflow for a connection is restarted and Airbyte decides to clean existing job state, forcing the current attempt to
FAILED
so it can be retried cleanly.[Google Ads sync error; HA not supported] Key points: • It is classified as an *internal transient Airbyte error*; the message explicitly says the sync should work on the next retry. • In some self‑hosted setups, this has been correlated with Temporal instability or misconfiguration, e.g. running Temporal with multiple replicas and/or high load on the Temporal metadata DB, which caused frequent workflow restarts and these transient failures.[HA not supported] • In other cases, users have seen similar messages when the Temporal workflow was restarted and Airbyte “cleaned” existing job state, causing the current attempt to be marked failed and requiring a retry.[Google Ads sync error] What you can do: 1. Retry the sync Since it’s transient, simply re‑running the sync often succeeds. 2. Check your Temporal deployment (self‑hosted) ◦ Ensure Temporal is healthy and not frequently restarting. ◦ If you are running multiple Temporal replicas for HA, be aware that this is currently not supported and can cause exactly these transient internal errors; some users resolved it by reducing Temporal to a single replica.[HA not supported] 3. If errors persist ◦ Inspect platform logs around the time of the error for Temporal or DB issues. ◦ Consider upgrading Airbyte if you’re on an older version, as suggested in similar cases.[Google Ads sync error] If you share your deployment type (Cloud vs OSS, version, and how Temporal is deployed), I can help you map these notes more concretely to your setup based on the available sources. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
k
@kapa.ai OSS version and below is temporal values.yaml temporal: enabled: true # -- The number of temporal replicas to deploy replicaCount: 1 image: # -- The temporal image repository to use repository: temporalio/auto-setup # -- The pull policy for the temporal image pullPolicy: IfNotPresent # -- The temporal image tag to use tag: "1.23.0" service: # -- The Kubernetes Service Type type: ClusterIP # -- The temporal port and exposed kubernetes port port: 7233 # -- Add extra annotations to the temporal pod podAnnotations: {} # -- Add extra labels to the temporal pod podLabels: {} # -- Security context for the container podSecurityContext: # gid=1000(temporal) fsGroup: 1000 containerSecurityContext: allowPrivilegeEscalation: false runAsNonRoot: true # uid=1000(temporal) runAsUser: 1000 # gid=1000(temporal) runAsGroup: 1000 readOnlyRootFilesystem: false capabilities: drop: ["ALL"] seccompProfile: type: RuntimeDefault extraInitContainers: [] livenessProbe: # -- Enable livenessProbe on the temporal enabled: false readinessProbe: # -- Enable readinessProbe on the temporal enabled: false # -- Node labels for temporal pod assignment, see https://kubernetes.io/docs/user-guide/node-selection/ nodeSelector: role: airbyte_nodes # -- Tolerations for temporal pod assignment, see https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/ tolerations: - key: "airbyte_nodes" operator: "Equal" value: "true" effect: "NoSchedule" affinity: {} extraEnv: [] extraVolumeMounts: [] extraVolumes: [] resources: limits: cpu: 400m memory: 1Gi requests: memory: 512Mi cpu: 200m extraContainers: []
k
Based on the knowledge sources, this error is almost always linked to Temporal instability or overload, not to the connector itself. You are already running
replicaCount: 1
, which is good: running multiple Temporal replicas with the current Airbyte Helm chart is known to cause exactly these transient internal errors and high DB load, and the recommended workaround is to keep Temporal at a single replica until proper clustering support exists.[HA temporal] Given that, the remaining likely causes are: 1. Temporal DB (PostgreSQL) under stress or slow When Temporal cannot reliably persist workflow state, workflows get restarted/cleaned up and Airbyte marks the attempt as failed with the message you’re seeing.[0‑records sync] The recommended checks are: ◦ CPU < ~70% ◦ Sufficient memory ◦ Active connections vs
max_connections
◦ Disk I/O latency / IOPS limits ◦ Slow queries / lock contention[0‑records sync] 2. Temporal pod under‑resourced Several issues show Temporal being “sensitive to delays and timeouts when interacting with the database”, and resource‑constrained Temporal pods or noisy DB can trigger context‑deadline / persistence problems.[Temporal deadline; 0‑records sync] Your current limits:
Copy code
limits:
     cpu: 400m
     memory: 1Gi
   requests:
     cpu: 200m
     memory: 512Mi
may be tight if you run many concurrent syncs. 3. High number of concurrent syncs Overwhelming Temporal (and its DB) with many workflows at once is explicitly called out as a cause of persistence failures and orphaned workflows.[0‑records sync] Concrete actions you can take All of these are directly recommended in the linked issues: 1. Check Temporal DB health (metrics & logs) and scale it up if you see high CPU, memory pressure, connection saturation, or disk I/O bottlenecks.[0‑records sync] 2. Increase Temporal pod resources For example, try bumping to something like (exact values depend on your cluster capacity, but the direction is to give Temporal more headroom):
Copy code
temporal:
     resources:
       requests:
         cpu: 500m
         memory: 1Gi
       limits:
         cpu: 1
         memory: 2Gi
(This is a suggestion based on the pattern in the issues; the exact numbers are not specified in the sources.) 3. Control concurrency Temporarily reduce the number of concurrent syncs if you’re running many at once, and see if the error rate drops.[0‑records sync] 4. Keep Temporal at
replicaCount: 1
You’re already doing this; keep it that way. The chart’s
replicaCount
option is known to be misleading because proper Temporal clustering (membership, discovery ports) is not wired up yet, and multiple replicas have caused DB load spikes and transient internal errors until users went back to 1 replica.[HA temporal] If, after increasing Temporal/DB resources and moderating concurrency, you still see frequent “internal transient Airbyte error” messages, the knowledge sources don’t provide more specific tuning parameters; at that point, capturing Temporal + DB logs around a failing attempt and opening a GitHub issue would be the next step. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
k
@kapa.ai Please check below logs {"level":"info","ts":"2026-02-23T051951.293Z","msg":"Failing the workflow task.","shard-id":4,"address":"*...*:7234","component":"history-engine","value":"UnhandledCommand","wf-id":"connection_manager_2dc1623b-c193-47e1-adb9-49cf2b73cabf","wf-run-id":"9ae8ccb9-74cf-4fbb-a6eb-3a47a3a9494a","wf-namespace-id":"03f2fb62-44c2-49a7-af1f-81ef2fbbacb6","logging-call-at":"workflow_task_handler_callbacks.go:606"} {"level":"info","ts":"2026-02-23T051951.424Z","msg":"history client encountered error","service":"frontend","error":"UnhandledCommand","service-error-type":"serviceerror.InvalidArgument","logging-call-at":"metric_client.go:104"} {"level":"info","ts":"2026-02-23T051957.241Z","msg":"history client encountered error","service":"matching","error":"Workflow task already started.","service-error-type":"serviceerror.TaskAlreadyStarted","logging-call-at":"metric_client.go:104"} {"level":"error","ts":"2026-02-23T052002.051Z","msg":"Update workflow execution operation failed.","shard-id":1,"address":"*...*:7234","wf-namespace-id":"03f2fb62-44c2-49a7-af1f-81ef2fbbacb6","wf-id":"connection_manager_108b4246-533f-4795-8030-c9ab79b27faf","wf-run-id":"89442327-466d-4982-b360-803931496caa","store-operation":"update-wf-execution","error":"context deadline exceeded","logging-call-at":"transaction_impl.go:511","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/service/history/workflow.updateWorkflowExecution\n\t/home/builder/temporal/service/history/workflow/transaction_impl.go:511\ngo.temporal.io/server/service/history/workflow.(*TransactionImpl).UpdateWorkflowExecution\n\t/home/builder/temporal/service/history/workflow/transaction_impl.go:189\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionWithNew\n\t/home/builder/temporal/service/history/workflow/context.go:623\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).UpdateWorkflowExecutionAsActive\n\t/home/builder/temporal/service/history/workflow/context.go:484\ngo.temporal.io/server/service/history/api.UpdateWorkflowWithNew\n\t/home/builder/temporal/service/history/api/update_workflow_util.go:113\ngo.temporal.io/server/service/history/api.GetAndUpdateWorkflowWithNew\n\t/home/builder/temporal/service/history/api/update_workflow_util.go:59\ngo.temporal.io/server/service/history/api/recordactivitytaskstarted.Invoke\n\t/home/builder/temporal/service/history/api/recordactivitytaskstarted/api.go:56\ngo.temporal.io/server/service/history.(*historyEngineImpl).RecordActivityTaskStarted\n\t/home/builder/temporal/service/history/history_engine.go:476\ngo.temporal.io/server/service/history.(*Handler).RecordActivityTaskStarted\n\t/home/builder/temporal/service/history/handler.go:315\ngo.temporal.io/server/api/historyservice/v1._HistoryService_RecordActivityTaskStarted_Handler.func1\n\t/home/builder/temporal/api/historyservice/v1/service_grpc.pb.go:1371\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:63\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:143\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:67\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1203\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/rate_limit.go:88\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1203\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:165\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1203\ngo.temporal.io/server/service.GrpcServerOptionsProvider.getUnaryInterceptors.NewServerMetricsTrailerPropagatorInterceptor.func4\n\t/home/builder/temporal/common/metrics/grpc.go:113\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1203\ngo.temporal.io/server/service.GrpcServerOptionsProvider.getUnaryInterceptors.NewServerMetricsContextInjectorInterceptor.func3\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1203\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.47.0/interceptor.go:326\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1203\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:145\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1194\ngo.temporal.io/server/api/historyservice/v1._HistoryService_RecordActivityTaskStarted_Handler\n\t/home/builder/temporal/api/historyservice/v1/service_grpc.pb.go:1373\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1386\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1797\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1027"} {"level":"error","ts":"2026-02-23T052002.421Z","msg":"Fail to process task","shard-id":4,"address":"*...*:7234","component":"transfer-queue-processor","wf-namespace-id":"03f2fb62-44c2-49a7-af1f-81ef2fbbacb6","wf-id":"sync_58192","wf-run-id":"8a304a8c-006e-4475-8ee3-47101d0bbf2e","queue-task-id":454091303,"queue-task-visibility-timestamp":"2026-02-23T052001.161Z","queue-task-type":"TransferWorkflowTask","queue-task":{"NamespaceID":"03f2fb62-44c2-49a7-af1f-81ef2fbbacb6","WorkflowID":"sync_58192","RunID":"8a304a8c-006e-4475-8ee3-47101d0bbf2e","VisibilityTimestamp":"2026-02-23T052001.161351886Z","TaskID":454091303,"TaskQueue":"1@airbyte-prod-airbyte-worker-7dc9469dcb-jsppx:beb104b6-ceb7-441a-827c-b81dfdaae9d3","ScheduledEventID":35,"Version":0},"wf-history-event-id":35,"error":"context deadline exceeded","unexpected-error-attempts":1,"lifecycle":"ProcessingFailed","logging-call-at":"lazy_logger.go:68","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:156\ngo.temporal.io/server/common/log.(*lazyLogger).Error\n\t/home/builder/temporal/common/log/lazy_logger.go:68\ngo.temporal.io/server/service/history/queues.(*executableImpl).HandleErr\n\t/home/builder/temporal/service/history/queues/executable.go:421\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask.func1\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:224\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:117\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:143\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:118\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).executeTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:233\ngo.temporal.io/server/common/tasks.(*FIFOScheduler[...]).processTask\n\t/home/builder/temporal/common/tasks/fifo_scheduler.go:211"} {"level":"info","ts":"2026-02-23T052003.264Z","msg":"history client encountered error","service":"matching","error":"Workflow task already started.","service-error-type":"serviceerror.TaskAlreadyStarted","logging-call-at":"metric_client.go:104"} {"level":"info","ts":"2026-02-23T052003.359Z","msg":"Duplicate activity retry timer task","shard-id":1,"address":"*...*:7234","component":"timer-queue-processor","wf-id":"connection_manager_cb1fc848-e531-44a3-a27f-f0651d8f7ea1","wf-run-id":"0e32ca89-bd4d-4068-8816-44bc59e44705","wf-namespace-id":"03f2fb62-44c2-49a7-af1f-81ef2fbbacb6","wf-scheduled-event-id":6,"attempt":2,"xdc-failover-version":0,"timer-task-status":8,"schedule-attempt":2,"logging-call-at":"timer_queue_active_task_executor.go:461"}
k
The knowledge sources only partially match what you’re seeing, so I can’t give a complete root cause, but I can relate your logs to the known patterns. From the linked GitHub issue, the key recurring Temporal errors in Airbyte OSS are: •
Workflow task already started.
(
serviceerror.TaskAlreadyStarted
) •
context deadline exceeded
during
UpdateWorkflowExecution
/ task processing These appear when Temporal is under load or slow to persist workflow state, and result in messages like “context deadline exceeded on EKS” and internal transient Airbyte errors.[Temporal deadline] In your logs: •
UnhandledCommand
and
Workflow task already started.
are Temporal history/matching errors around workflow task handling. •
Update workflow execution operation failed ... error:"context deadline exceeded"
and
Fail to process task ... error:"context deadline exceeded"
match the same class of persistence/timeout problems described in the issue.[Temporal deadline] In the referenced case, these Temporal timeouts led to workflows being marked as failed/stuck and retried, which surfaced in Airbyte as internal transient errors; the discussion points to Temporal/DB performance or stability as the underlying problem, not a connector bug.[Temporal deadline] However, the knowledge sources: • Do not mention
UnhandledCommand
specifically. • Do not provide a precise fix beyond investigating Temporal health, DB performance, and timeouts. So, based strictly on the sources: • Your
context deadline exceeded
and
Workflow task already started
errors are consistent with the known Temporal timeout/instability issue in Airbyte OSS.[Temporal deadline] • The sources do not give additional, concrete remediation steps for
UnhandledCommand
beyond treating it as part of the same Temporal workflow failure pattern. If you need more targeted guidance (e.g., specific Temporal/DB tuning or how to handle
UnhandledCommand
), that information is not present in the provided knowledge, and you’d likely need to open a new GitHub issue with these exact logs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.