Hey, @kapa.ai what is good practices to scale temporal on self hosted kubernetes setup? Vertically with adding more resources and configuring ``SQL_MAX_IDLE_CONNS` capped by

SQL_MAX_CONNS

, or horizontally adding more pods?

The knowledge sources only give partial guidance here, so I’ll focus on what’s explicitly documented. From the scaling docs, the main recommendation is to ensure your Kubernetes cluster can schedule enough pods for Airbyte jobs and to tune Temporal’s DB connection settings so it doesn’t overload Postgres. [Scaling guide] For Temporal specifically: • Temporal uses multiple connection pools, so the effective number of DB connections can be 4–5× what you might expect from a single pool. [Scaling guide] •

SQL_MAX_IDLE_CONNS

is capped by

SQL_MAX_CONNS

, so if you increase idle connections you must also increase

SQL_MAX_CONNS

. [Scaling guide] The docs do not explicitly recommend horizontal vs vertical scaling of Temporal pods. They only highlight: • Make sure the Temporal DB can handle the number of connections (by tuning

SQL_MAX_CONNS

/

SQL_MAX_IDLE_CONNS

and DB resources). [Scaling guide] • Monitor Temporal DB health (CPU, memory, connections, IOPS) and scale the database instance if it’s under load. [GitHub issue 70984] Given that, the documented best practices are: 1. Start with DB-side scaling and connection tuning ◦ Ensure Postgres for Temporal has enough CPU, memory, IOPS. [GitHub issue 70984] ◦ Adjust

SQL_MAX_CONNS

and

SQL_MAX_IDLE_CONNS

so Temporal doesn’t exhaust DB connections, remembering multiple pools can multiply the effective count. [Scaling guide] 2. Then adjust Temporal server resources ◦ Increase CPU/memory limits/requests for Temporal server pods if they’re constrained. [GitHub issue 70984] 3. Horizontal vs vertical The sources don’t clearly state whether you should add more Temporal pods vs scaling a single pod up. They only warn about DB connections and resource pressure. So I can’t confidently say which is “preferred” in Airbyte’s guidance beyond: whichever approach you choose, ensure the DB connection limits and resources are sized accordingly. If you share how many concurrent syncs you’re targeting and your current Temporal DB metrics (CPU, memory, connections), you can then size `SQL_MAX_CONNS`/`SQL_MAX_IDLE_CONNS` and DB resources to match. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

What about server replicas, @kapa.ai should we spin 2 servers replicas (pods), or same for workload launcher, etc?

Airbyte’s docs don’t give Temporal‑specific replica guidance, but they do cover replicas for core Airbyte services and HA patterns. Server & worker replicas For high availability, the recommendation is to run at least 2 replicas of the

server

and

worker

components:

worker:
  replicaCount: 2  

server:
  replicaCount: 2

This is explicitly called out as the suggested HA setup. [HA scaling] You can also increase

worker.replicaCount

further if you want to run more concurrent syncs, together with tuning

MAX_SYNC_WORKERS

/

MAX_CHECK_WORKERS

. [Enterprise scaling] Workload launcher / other components The “What to scale” guides focus on: • Ensuring the Kubernetes cluster can schedule enough job pods (rough rule: up to

2 x

number-of-possible-concurrent-connections pods). [Operator scaling] • Memory and disk sizing, and Temporal DB connection settings (

SQL_MAX_IDLE_CONNS

capped by

SQL_MAX_CONNS

). [Operator scaling] They do not provide explicit replica-count recommendations for the workload launcher or Temporal itself, beyond the general HA pattern above and DB‑connection tuning. So I can’t confidently say “run N replicas of the workload launcher/Temporal” based on the docs alone. Practical takeaway from the docs • For HA: set

server.replicaCount: 2

and

worker.replicaCount: 2

. [HA scaling] • For throughput: increase

worker.replicaCount

and the

MAX_SYNC_WORKERS

/

MAX_CHECK_WORKERS

limits, while ensuring your cluster and Temporal DB can handle the extra load. [Enterprise scaling] • For Temporal DB: tune

SQL_MAX_CONNS

and

SQL_MAX_IDLE_CONNS

carefully, since multiple pools can multiply effective connections. [Operator scaling] Beyond that, the knowledge sources don’t specify more detailed replica strategies for Temporal or the workload launcher. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

What about configuring temporal ratelimiter?

[io.airbyte.cron.jobs.SelfHealTemporalWorkflows@64158e2a] RESOURCE_EXHAUSTED: namespace rate limit exceeded

@kapa.ai

The knowledge sources only partially cover this, and they do not document how to configure Temporal’s own rate limiter (e.g., per-namespace RPS). They only show how the problem surfaces in Airbyte and what has been improved on Airbyte’s side. What we do know from the docs and issues: • The error you’re seeing is exactly the Temporal error:

RESOURCE_EXHAUSTED: namespace rate limit exceeded

. [Issue 48879; Issue 68115] • Airbyte maintainers linked this to Temporal configuration and merged “better Temporal defaults” into Airbyte around v1.5.0. [Issue 48879] • The operator guide calls out Temporal DB behavior and connection limits, but not rate‑limit knobs: ◦ Temporal maintains multiple idle connection pools. ◦

SQL_MAX_IDLE_CONNS

is capped by

SQL_MAX_CONNS

. ◦ Because of multiple pools, you can end up with 4–5× more DB connections than expected. [Scaling Airbyte] From these, the documented levers you can safely use in Airbyte are: 1. Upgrade Airbyte to a version with improved Temporal defaults The fix mentioned in the issue is available from v1.5.0 onward. If you’re on an older version, upgrading is the first recommended step. [Issue 48879] 2. Ensure Temporal’s DB is not the bottleneck ◦ Increase DB resources (CPU, RAM, IOPS) if it’s saturated. ◦ Tune

SQL_MAX_CONNS

and

SQL_MAX_IDLE_CONNS

so Temporal has enough connections but doesn’t overwhelm Postgres, remembering the 4–5× multiplier from multiple pools. [Scaling Airbyte] 3. Reduce load on Temporal if needed ◦ Limit how many concurrent syncs you run (via worker counts / job concurrency) so Temporal receives fewer workflow calls per second. [Scaling Airbyte] What is not documented in these sources: • Any Airbyte‑specific guidance on changing Temporal’s namespace rate‑limit configuration (e.g., Temporal server flags or dynamic config for RPS). • Exact values to set for rate limits or how to override them in the bundled

temporalio/auto-setup

image. So, based on the available information: • You can address many

RESOURCE_EXHAUSTED

issues by: ◦ Upgrading Airbyte to pick up better Temporal defaults. ◦ Right‑sizing the Temporal DB and its connection limits. ◦ Reducing Airbyte’s concurrency so Temporal receives fewer requests. • The knowledge sources do not explain how to directly tune Temporal’s rate limiter in the Airbyte Helm chart or deployment, so I can’t give a precise, supported config snippet for that. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.