@kapa.ai Our

airbyte-temporal

pod in kubernetes deployment regularly (approx each 5 days) reaches its CPU limit quota (CPU utilisation to 100%) and blocks triggering the next workloads. What can be the root cause of this behaviour?

Based on the knowledge sources, there are several known root causes for the

airbyte-temporal

pod hitting CPU limits and blocking workloads: Known Root Causes 1. Temporal Rate Limiting / Resource Exhaustion One documented pattern is Temporal hitting rate limits (

RESOURCE_EXHAUSTED

) when too many workflows are being processed concurrently. This can cause CPU spikes as Temporal struggles to keep up. [Github Issue] 2. Workflow Thread Leaks A known bug where workflow threads can't be destroyed in time leads to a workflow cache leak. Over time, these stuck/leaked workflows accumulate and consume increasing CPU resources until the pod is saturated. [Github Issue] 3. Cron Hitting Rate Limits During Cleanup The

cron

component can continuously hit Temporal rate limits while trying to clean up workflows, driving up CPU usage on the Temporal pod. [Github Issue] 4. High Number of Concurrent Syncs Running a large number of concurrent syncs (e.g., 500+ syncs every two hours) puts significant pressure on Temporal, leading to CPU exhaustion and blocked workload launches. [Github Issue] 5. Under-provisioned Resources Insufficient CPU limits on the Temporal pod itself can cause it to throttle under normal load, manifesting as

DEADLINE_EXCEEDED

errors and blocked syncs. [Forum Post; Forum Post] Recommendations • Increase Temporal pod CPU limits in your Helm values. • Check your Airbyte version — fixes for Temporal defaults and cron rate-limiting were merged and released in v1.5.0 of the platform. If you're on an older version, upgrading may resolve the issue. [Github Issue] • Reduce concurrent syncs by setting

MAX_SYNC_WORKERS

and staggering sync schedules to reduce Temporal load. [Scaling Airbyte] • Monitor Temporal metrics such as

persistence_latency

and

persistence_errors

to identify the specific bottleneck. [Github Issue] • Tune Temporal DB connections via

SQL_MAX_IDLE_CONNS

and

SQL_MAX_CONNS

if database connection pressure is contributing. [Scaling Airbyte] The ~5-day cycle suggests a gradual accumulation (e.g., thread/workflow leaks or growing backlog) rather than an immediate overload, which aligns most closely with the workflow thread leak pattern described above. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai we use airbyte 2.0.1

Thanks for sharing your version. You're running Airbyte 2.0.1, which is a relatively recent release in the 2.x line. Based on the knowledge sources, here's what's most relevant for your version: Temporal CPU Resource Configuration (v2.x) In Airbyte 2.x Helm charts, you can explicitly set CPU/memory resources for the Temporal pod in your `values.yaml`: [Values.yaml reference]

temporal:
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      cpu: 2000m      # Increase this if hitting CPU limits
      memory: 2Gi

Concurrent Sync Limits If you're running many concurrent syncs, you can limit them to reduce pressure on Temporal: [Scaling Airbyte]

worker:
  maxSyncWorkers: 5    # Adjust based on your workload
  maxCheckWorkers: 5

Temporal DB Connections If your Temporal database is under stress, tuning idle connections may help. The default is

20

, but Temporal can create 4–5x more connections than expected due to multiple pools. Adjust via

SQL_MAX_IDLE_CONNS

and

SQL_MAX_CONNS

environment variables. [Scaling Airbyte] Key Recommendation The ~5-day cycle strongly suggests a gradual accumulation (workflow thread leaks or growing backlog) rather than immediate overload. The knowledge sources do not contain specific patch notes for v2.0.1 addressing this, so I'd recommend: 1. Increasing the Temporal pod CPU limit in

values.yaml

2. Monitoring concurrent sync counts and reducing them if high 3. Checking Temporal DB health (CPU, connections, query latency) If the issue persists, consider opening a GitHub issue referencing your version, as the knowledge sources indicate similar issues in v1.8.x were escalated for engineering investigation. [Github Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai could you explain what's temporal responsible for?

In Airbyte's architecture, Temporal serves as the orchestration engine. Specifically: • Temporal Service (

airbyte-temporal

) manages the scheduling and sequencing of task queues and workflows. [Architecture overview] It works in conjunction with the Worker component, which reads from those task queues and executes the connection scheduling and sequencing logic. Why Temporal was chosen Temporal solves two major challenges when orchestrating hundreds to thousands of jobs simultaneously: 1. Scaling state management — it serializes a job's current runtime memory into a database, so the entire state of a job is stored persistently. 2. Proper queue management — it ensures jobs are assigned and tracked correctly. Because the full job state is persisted, it becomes trivial to recover from failures and easy to determine if a job was assigned correctly. [Tech Stack] In the context of your issue This is why a CPU-saturated Temporal pod directly blocks workload triggering — if Temporal can't process its task queues, the Worker has nothing to consume, and no new sync jobs get launched. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai what metrics we can monitor related to temporal?

The knowledge sources don't contain detailed information specifically about Temporal-internal metrics in the context of Airbyte's deployment. However, here's what is mentioned: Temporal Metrics to Monitor The knowledge sources reference these specific Temporal-related metrics to watch for health issues: •

persistence_latency

— should be kept low (< 100ms) •

persistence_errors

— should be zero • Workflow task queue backlog — indicates if Temporal is falling behind [Github Issue] Airbyte-Level Metrics (via OpenTelemetry) If you're on Airbyte Self-Managed Enterprise, you can send metrics to a monitoring tool (e.g., Datadog) via OpenTelemetry. The available Airbyte metrics include sync-level data such as: [OpenTelemetry metrics] These are published every minute and come from the

airbyte-worker

service, which interacts directly with Temporal. Limitation The knowledge sources do not provide a comprehensive list of Temporal-native metrics (e.g., from Temporal's own metrics endpoint). For those, I'd recommend consulting Temporal's official documentation directly, as that is outside the scope of what the Airbyte knowledge sources cover. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements