https://linen.dev logo
Join Slack
Powered by
o
loading...
h
Hey can you check if all the pods are running (temporal, server, scheduler). Also if you can share the logs of them
n
web app, server and worker are not getting started
for server and worker both shows waiting for default namespace to be inti...
HERE ARE THE TEMPORAL LOGS
Copy code
{"level":"info","ts":"2022-03-22T05:41:34.270Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/2","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:209"}
{"level":"info","ts":"2022-03-22T05:41:35.270Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Starting","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/3","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:206"}
{"level":"info","ts":"2022-03-22T05:41:35.270Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/3","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:209"}
{"level":"info","ts":"2022-03-22T05:41:35.271Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Starting","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:206"}
{"level":"info","ts":"2022-03-22T05:41:35.271Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/1","wf-task-queue-type":"Workflow","logging-call-at":"matchingEngine.go:209"}
{"level":"info","ts":"2022-03-22T05:41:35.273Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Starting","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/1","wf-task-queue-type":"Activity","logging-call-at":"matchingEngine.go:206"}
{"level":"info","ts":"2022-03-22T05:41:35.273Z","msg":"none","service":"matching","component":"matching-engine","lifecycle":"Started","wf-task-queue-name":"/_sys/temporal-sys-add-search-attributes-task-queue/1","wf-task-queue-type":"Activity","logging-call-at":"matchingEngine.go:209"}
{"level":"error","ts":"2022-03-22T05:41:36.719Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/1","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:41:37.261Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"airbyte-temporal-59966ff5bd-xjvqq:f1f9118f-71d8-4369-922e-bfdcb51f4f14","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:41:37.424Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-batcher-taskqueue","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:41:40.428Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"temporal-sys-processor-parent-close-policy","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:42:13.150Z","msg":"Membership upsert failed.","service":"matching","error":"Persistence Max QPS Reached.","logging-call-at":"rpMonitor.go:276","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/temporal/common/membership/rpMonitor.go:276|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/temporal/common/membership/rpMonitor.go:276>"}
{"level":"error","ts":"2022-03-22T05:42:17.112Z","msg":"Error refreshing namespace cache","service":"matching","error":"Persistence Max QPS Reached.","logging-call-at":"namespaceCache.go:414","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/temporal/common/cache/namespaceCache.go:414|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/common/cache.(*namespaceCache).refreshLoop\n\t/temporal/common/cache/namespaceCache.go:414>"}
{"level":"error","ts":"2022-03-22T05:42:36.502Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/2","wf-task-queue-type":"Activity","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:42:37.259Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-tq-scanner-taskqueue-0/1","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:42:37.260Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"airbyte-temporal-59966ff5bd-xjvqq:35c3ad98-fe23-4fa4-9b33-46ae76a31a64","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:42:37.261Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"airbyte-temporal-59966ff5bd-xjvqq:f1f9118f-71d8-4369-922e-bfdcb51f4f14","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:42:37.404Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-processor-parent-close-policy/2","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
{"level":"error","ts":"2022-03-22T05:42:37.859Z","msg":"Persistent store operation failure","service":"matching","component":"matching-engine","wf-task-queue-name":"/_sys/temporal-sys-batcher-taskqueue/2","wf-task-queue-type":"Workflow","store-operation":"update-task-queue","error":"Persistence Max QPS Reached.","logging-call-at":"taskReader.go:183","stacktrace":"<http://go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183|go.temporal.io/server/common/log.(*zapLogger).Error\n\t/temporal/common/log/zap_logger.go:143\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/temporal/service/matching/taskReader.go:183>"}
@Harshith (Airbyte) I was unable to find starting server in pod logs for temporal service, I am not sure how it is configured on kubernetes
@Harshith (Airbyte) is it possible that due to some sort of access issue for storage namespace may not get initialised ?
h
Can you also share the server and scheduler logs?
n
nevermind resolved
Open in Slack
PreviousNext
Powered by