https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> What does this error from temporal ...
# replication-ask-ai
g
@kapa.ai What does this error from temporal mean?
Copy code
{
  "level": "error",
  "ts": "2026-03-19T02:08:31.299Z",
  "msg": "Persistent fetch operation Failure",
  "shard-id": 1,
  "address": "0.0.0.0:7234",
  "wf-namespace-id": "26cd0618-b217-45c1-ab3e-a8f2fddb9598",
  "wf-id": "connection_manager_c21e1c31-d7b2-47ae-bf3d-914c02c3360d",
  "wf-run-id": "9df9d9b1-e789-4daf-9520-ef702999e7a2",
  "store-operation": "get-wf-execution",
  "error": "shard status unknown",
  "logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:476",
  "stacktrace": "<http://go.temporal.io/server/common/log.(*zapLogger).Error|go.temporal.io/server/common/log.(*zapLogger).Error>\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:154\ngo.temporal.io/server/service/history/workflow.getWorkflowExecution\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/transaction_impl.go:476\ngo.temporal.io/server/service/history/workflow.(*ContextImpl).LoadMutableState\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/workflow/context.go:261\ngo.temporal.io/server/service/history/api.(*WorkflowConsistencyCheckerImpl).getWorkflowLease\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/api/consistency_checker.go:248\ngo.temporal.io/server/service/history/api.(*WorkflowConsistencyCheckerImpl).getWorkflowLeaseImpl\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/api/consistency_checker.go:144\ngo.temporal.io/server/service/history/api.(*WorkflowConsistencyCheckerImpl).GetWorkflowLease\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/api/consistency_checker.go:115\ngo.temporal.io/server/service/history/api.GetAndUpdateWorkflowWithNew\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/api/update_workflow_util.go:48\ngo.temporal.io/server/service/history/api/recordactivitytaskstarted.Invoke\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/api/recordactivitytaskstarted/api.go:62\ngo.temporal.io/server/service/history.(*historyEngineImpl).RecordActivityTaskStarted\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/history_engine.go:534\ngo.temporal.io/server/service/history.(*Handler).RecordActivityTaskStarted\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/history/handler.go:342\ngo.temporal.io/server/api/historyservice/v1._HistoryService_RecordActivityTaskStarted_Handler.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/api/historyservice/v1/service_grpc.pb.go:1677\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/retry.go:62\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:89\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/retry.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/rate_limit.go:88\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/interceptor/telemetry.go:197\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/service.GrpcServerOptionsProvider.getUnaryInterceptors.NewServerMetricsTrailerPropagatorInterceptor.func6\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:112\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/service.GrpcServerOptionsProvider.getUnaryInterceptors.NewServerMetricsContextInjectorInterceptor.func5\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/grpc.go:65\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1217\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/rpc/grpc.go:157\ngoogle.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1208\ngo.temporal.io/server/api/historyservice/v1._HistoryService_RecordActivityTaskStarted_Handler\n\t/home/runner/work/docker-builds/docker-builds/temporal/api/historyservice/v1/service_grpc.pb.go:1679\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1405\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1815\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\t/home/runner/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1035"
}
k
This error from Temporal's history service indicates a "shard status unknown" condition — meaning the Temporal history shard responsible for the 
connection_manager
workflow could not be accessed when trying to fetch workflow execution state (
store-operation: get-wf-execution
). What It Means The key details from the error: • 
msg: "Persistent fetch operation Failure"
 — Temporal failed to read workflow state from its persistence layer (database). • 
error: "shard status unknown"
 — The history shard's status could not be determined, preventing Temporal from loading the mutable state of the workflow. • 
store-operation: "get-wf-execution"
 — This happened during a read (not a write), specifically when trying to record an activity task as started. • 
wf-id: "connection_manager_..."
 — This affects an Airbyte connection manager workflow, which orchestrates sync jobs. Likely Causes Based on similar issues reported in the Airbyte community, this type of Temporal persistence failure is typically a self-hosted infrastructure issue, not an Airbyte platform bug. Common causes include: [sync pulling 0 records] 1. Temporal database (PostgreSQL) under stress — high CPU, memory pressure, slow disk I/O, or too many connections. 2. Multiple Temporal replicas competing for shard ownership — running more than one Temporal server replica can cause shard conflicts and instability. [High availability not supported] 3. Too many concurrent syncs overwhelming the Temporal cluster. 4. Temporal pod resource constraints — insufficient CPU/memory allocated to the Temporal pod. Recommended Actions 1. Check your Temporal database health — review CPU, memory, disk I/O, and connection counts on the PostgreSQL instance backing Temporal. 2. Ensure only one Temporal replica is running — high availability (multiple replicas) is not supported and can cause shard ownership conflicts. [High availability not supported] 3. Review Temporal pod resource allocation — increase CPU/memory limits if the pod is resource-constrained. 4. Check for excessive concurrent syncs that may be overwhelming the cluster. 5. Review Temporal and database logs for additional context around the time of the failure. The downstream effect of this error is that Airbyte workflows may fail silently or syncs may appear to complete with 0 records and no error, because Temporal can no longer track the workflow state. [sync pulling 0 records] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
5 Views
Open in Slack
PreviousNext
Powered by