Dmytro Shamenko
03/03/2026, 11:31 AMairbyte-cron-6cbc86fbb-plhxg 2026-03-03 11:30:52,487 [scheduled-executor-thread-1] ERROR i.m.s.DefaultTaskExceptionHandler(handle):47 - Error invok ā
ā ing scheduled task for bean [io.airbyte.cron.jobs.SelfHealTemporalWorkflows@2197685b] RESOURCE_EXHAUSTED: namespace rate limit exceeded
where the hack an issue with this temporal?
Name: temporal-dynamicconfig-custom
Namespace: airbyte
Labels: <http://kustomize.toolkit.fluxcd.io/name=airbyte|kustomize.toolkit.fluxcd.io/name=airbyte>
<http://kustomize.toolkit.fluxcd.io/namespace=flux-system|kustomize.toolkit.fluxcd.io/namespace=flux-system>
Annotations: <none>
Data
====
development.yaml:
----
# Copied from airbyte-temporal-dynamicconfig chart defaults.
# RPS limits bumped to prevent RESOURCE_EXHAUSTED / namespace rate limit exceeded errors.
# persistenceMaxQPS: 3000 -> 9000
# frontend.rps, frontend.namespaceRPS, matching.rps, history.rps -> 76800 (validated in load testing)
frontend.enableClientVersionCheck:
- value: true
constraints: {}
history.persistenceMaxQPS:
- value: 9000
constraints: {}
frontend.persistenceMaxQPS:
- value: 9000
constraints: {}
frontend.rps:
- value: 76800
constraints: {}
frontend.namespaceRPS:
- value: 76800
constraints: {}
frontend.historyMgrNumConns:
- value: 30
constraints: {}
frontend.throttledLogRPS:
- value: 20
constraints: {}
frontend.enableUpdateWorkflowExecution:
- value: true
frontend.enableUpdateWorkflowExecutionAsyncAccepted:
- value: true
history.rps:
- value: 76800
constraints: {}
history.historyMgrNumConns:
- value: 50
constraints: {}
matching.rps:
- value: 76800
constraints: {}
system.advancedVisibilityWritingMode:
- value: "off"
constraints: {}
history.defaultActivityRetryPolicy:
- value:
InitialIntervalInSeconds: 1
MaximumIntervalCoefficient: 100.0
BackoffCoefficient: 2.0
MaximumAttempts: 0
history.defaultWorkflowRetryPolicy:
- value:
InitialIntervalInSeconds: 1
MaximumIntervalCoefficient: 100.0
BackoffCoefficient: 2.0
MaximumAttempts: 0
limit.blobSize.error:
- value: 15728640 # 15MB
constraints: {}
limit.blobSize.warn:
- value: 10485760 # 10MB
constraints: {}
BinaryData
====
Events: <none>kapa.ai
03/03/2026, 11:31 AMRESOURCE_EXHAUSTED: namespace rate limit exceeded error from SelfHealTemporalWorkflows is a known issue in Airbyte's cron service related to Temporal rate limiting. [rate limit issue]
Looking at your dynamic config, you've already bumped the main RPS limits significantly. However, there are a few things to check:
What might still be missing
Your config is missing some visibility-specific rate limit settings. Compare with a known working config from the community:
frontend.namespaceRPS.visibility:
- value: 100
constraints: {}
frontend.namespaceBurst.visibility:
- value: 150
constraints: {}
frontend.namespaceCount:
- value: 4096
constraints: {}
frontend.persistenceMaxQPS:
- value: 5000
constraints: {}
[working config example]
The error in the stack trace specifically points to listClosedWorkflowExecutions, which is a visibility query ā so the frontend.namespaceRPS.visibility limit is likely the one being hit, not the general frontend.namespaceRPS. [stack trace reference]
Recommended actions
1. Add the missing visibility limits (frontend.namespaceRPS.visibility, frontend.namespaceBurst.visibility) to your ConfigMap.
2. Ensure you're on Airbyte v1.5.0+: A fix was merged that improved Temporal defaults and reduced how aggressively the cron service polls Temporal. The contributor confirmed: "both the better temporal defaults and the cron not running with all the components have been merged and should be available in our most recent release (1.5.0)". [fix confirmation]
3. Verify the ConfigMap is actually being mounted by the Temporal pod ā confirm the dynamic config file path is correctly referenced in the Temporal deployment.
The error being non-fatal (it's logged as ERROR but the cron continues) is expected behavior, but adding the visibility-specific limits should reduce or eliminate it. š 0 button š 0 button ā¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.