Bogdan
11/24/2025, 10:18 AMmax_check_workers=10 and max_sync_workers=10 in my Airbyte setup, but syncs are still running sequentially instead of in parallel. Has anyone encountered this issue before? What could be causing this behavior despite having these values set? Any suggestions would be greatly appreciated. @kapa.aikapa.ai
11/24/2025, 10:19 AMmax_check_workers=10 and `max_sync_workers=10`: 1. Worker limits are per worker pod, not global MAX_SYNC_WORKERS and MAX_CHECK_WORKERS (or the equivalent Helm worker.maxSyncWorkers, worker.maxCheckWorkers) are per Airbyte worker container limits, not a guarantee of global concurrency.[\[Configuring worker\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker) If you only have one worker pod and other constraints are low, you may still see just a few (or even one) sync at a time. 2. Overall submitter limit (SUBMITTER_NUM_THREADS) In K8s deployments, SUBMITTER_NUM_THREADS is a hard upper bound on total concurrent jobs in the system, regardless of MAX_*_WORKERS.[\[Scaling k8s parallelism\]](https://discuss.airbyte.io/t/826) If this is still at a low default, it can effectively serialize your syncs even though MAX_SYNC_WORKERS is 10. 3. Temporal worker ports can cap parallelism Each worker pod can only run as many jobs as it has Temporal worker ports exposed. The Helm chart exposes ports 9001–9040 by default, i.e., at most 40 parallel jobs per worker.[\[Configuring worker\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker) If TEMPORAL_WORKER_PORTS is misconfigured or too small relative to MAX_*_WORKERS, jobs can hang or appear serialized.[\[Scaling k8s parallelism\]](https://discuss.airbyte.io/t/826) 4. Replica count of worker pods To increase parallelism, you often also need multiple worker replicas, each able to run up to MAX_SYNC_WORKERS jobs.[\[Scaling k8s parallelism\]](https://discuss.airbyte.io/t/826) The scaling guides explicitly recommend increasing worker.replicaCount along with the max worker values.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits) 5. Cluster / resource pressure can effectively serialize jobs If your Kubernetes cluster doesn’t have enough CPU/memory to schedule connector pods, syncs can sit pending and run effectively one after another, even though Airbyte’s logical limits are higher.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs) Concrete checks / suggestions: - Verify SUBMITTER_NUM_THREADS is set high enough compared to your target concurrency.[\[Scaling k8s parallelism\]](https://discuss.airbyte.io/t/826) - Confirm worker.maxSyncWorkers / worker.maxCheckWorkers (or MAX_*_WORKERS envs) are actually applied to the running worker pods. - Ensure worker.replicaCount > 1 if you want higher global concurrency.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits) - Check that TEMPORAL_WORKER_PORTS covers at least the total concurrent jobs per worker.[\[Configuring worker\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker) - Look at your node capacity and pod scheduling (many pending orchestrator/read/write pods can mean cluster under‑provisioning).[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs) If you can share how you’ve set SUBMITTER_NUM_THREADS, worker.replicaCount, and your deployment type (Helm values vs env vars), I can help narrow which of these is most likely in your case.Bogdan
11/24/2025, 10:30 AMinfo and then changed it to debug, but I don't see any changes in the logs. There are no debug level log entries showing up. Does anyone know why this might be happening or what I might be missing? @kapa.aikapa.ai
11/24/2025, 10:30 AMkapa.ai
11/24/2025, 10:30 AMLOG_LEVEL Airbyte’s operator docs state that LOG_LEVEL defines the log level and defaults to INFO, and it must be set to one of the Log4J levels.[\[Configuring Airbyte\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#logging) If you changed a “global logging level” somewhere else (e.g., only in Helm values for a single component or in a config file that isn’t applied), it may not actually be changing the effective LOG_LEVEL env var of the running pods. 2. You must restart / redeploy components to pick up the new level The knowledge sources do not explicitly document hot‑reloading of log levels. This usually means that changing LOG_LEVEL in values or environment requires restarting the affected pods to take effect. The docs only describe LOG_LEVEL as a configuration parameter, not as something you can toggle at runtime.[\[Configuring Airbyte\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#logging) If you edited values (e.g., Helm) but didn’t re‑deploy or roll pods, the logs would stay at the previous level. 3. Beware of DEBUG in production Airbyte explicitly recommends turning off DEBUG logs for non‑testing use, because at scale they can overload the server pod and affect the deployment.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#debug-logs) This implies that when DEBUG is correctly enabled, you should indeed see significantly more log volume. If you see no difference, it strongly suggests the new level isn’t actually applied. 4. Per‑component vs “global” logging Some references (Helm values.yaml snippets, GitHub issues) show log levels being set on specific components like server.log.level: "DEBUG" or workload-launcher.log.level: "INFO".[\[Issue 59177](https://github.com/airbytehq/airbyte/issues/59177); \[Issue 60915](https://github.com/airbytehq/airbyte/issues/60915)] If you only changed one component’s level, you’ll see DEBUG logs only from that component, not from others (e.g., connectors, workers). 5. Connector / actor logs are protocol‑based Connector logs are emitted as AirbyteLogMessage objects wrapped in AirbyteMessage, with a level field such as INFO, DEBUG, etc.[\[AirbyteLogMessage](https://docs.airbyte.com/platform/understanding-airbyte/airbyte-protocol#airbytelogmessage)] If the connector code itself doesn’t emit any DEBUG‑level messages, you won’t see DEBUG lines even if the platform log level is DEBUG. Because the docs don’t provide a step‑by‑step example for switching from INFO to DEBUG and verifying it, there is some uncertainty about your exact setup. But based on the available information, the most likely causes are: - The LOG_LEVEL change is not actually applied to the running pods (no restart/redeploy, or changed in the wrong place). - You changed log level only for one component (e.g., server) and are looking at logs for another component that remains at INFO. - The code you’re observing simply doesn’t log at DEBUG level. I’d suggest: 1. Confirm which component’s values you changed (e.g., server.log.level, worker.log.level, etc.).[\[Issue 59177](https://github.com/airbytehq/airbyte/issues/59177)] 2. Verify in the pod’s environment that LOG_LEVEL is actually set to DEBUG for that component.[\[Configuring Airbyte\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#logging) 3. Restart or redeploy the affected pods after changing the value. 4. Check a component known to be noisy at DEBUG (e.g., server) and confirm increased log volume, keeping in mind the warning about not leaving DEBUG on in production.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#debug-logs)Bogdan
11/24/2025, 10:43 AMworker.replicaCount variable in the values.yml file for my Airbyte setup, but I’m not sure if it’s there. Can anyone confirm whether this variable is located in the values.yml file, or if it’s configured elsewhere? @kapa.aikapa.ai
11/24/2025, 10:43 AMkapa.ai
11/24/2025, 10:43 AMworker.replicaCount is a value that lives in the Airbyte Helm chart’s values.yaml and is used to control how many worker pods are deployed. The scaling docs and example configs show it explicitly under the worker section, for both Helm chart v1 and v2, for example: yaml worker: replicaCount: 2 [[Scaling Airbyte 1.8](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#concurrent-sync-limits); [Scaling Airbyte (latest)](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits)] If you don’t see it in your local values.yaml, it may just be omitted and falling back to the chart’s default. In that case you can add a worker: section with replicaCount yourself, or check the chart’s values.yaml on GitHub for the exact structure. [[Helm values ref](https://docs.airbyte.com/platform/deploying-airbyte/values)]Bogdan
11/24/2025, 2:57 PMfetch_size parameter. We have noticed that the number of records transferred in each batch is dynamically adjusted, but this is sometimes leading to incomplete data transfer — some tables are missing data, and not all rows are being transferred.
Is there a way to adjust the fetch_size manually for better control over how Airbyte handles these large data batches? Can this be configured in the Airbyte UI, or would it require modifying the integration's code? @kapa.aikapa.ai
11/24/2025, 2:57 PMkapa.ai
11/24/2025, 2:57 PMfetchSize mechanism, and this is not currently user‑configurable from the UI. From the shared discussions: - JDBC/source connectors fetch records in batches whose size is controlled by fetchSize. This was changed to be computed dynamically by the connector code.[[MySQL perf](https://discuss.airbyte.io/t/1092); [Sync performance](https://discuss.airbyte.io/t/1578)] - Users explicitly asked whether fetchSize can be configured; the answer from the team was “No, it’s not possible”, though the dynamic logic “should now be more clever.”[[Sync jobs speed up](https://discuss.airbyte.io/t/1168)] - For Postgres/Snowflake, support also confirms that fetchSize is adjusted dynamically and links performance concerns to an internal GitHub issue, rather than exposing a setting.[[Optimized Postgres](https://discuss.airbyte.io/t/1984); [Sync performance](https://discuss.airbyte.io/t/1578)] There is no mention in the docs or forum excerpts of: - A UI setting to override fetchSize. - A documented connector‑level config (YAML or JSON) that lets you set a custom fetchSize. So, based on the available information: - You cannot manually tune fetch_size via the Airbyte UI or standard connection settings. - Changing it would require modifying the connector implementation itself (i.e., editing the integration code / building a forked connector), but the exact code locations or patterns are not described in the provided sources. Given the issues you’re seeing (missing rows / incomplete transfer), those forum threads suggest: - Treat it as a bug / performance issue in the specific connector rather than something you can tune away with configuration.[[MySQL missing rows](https://github.com/airbytehq/airbyte/issues/53614); [MySQL perf](https://discuss.airbyte.io/t/1092)] If you share which source connector you’re using (MySQL, Postgres, Snowflake, etc.), the next best step is usually: 1. Capture and share full sync logs with Airbyte support/community so they can check how fetchSize is being computed in your runs.[[MySQL perf](https://discuss.airbyte.io/t/1092); [Sync jobs speed up](https://discuss.airbyte.io/t/1168)] 2. Open or link to a GitHub issue for that connector if rows are definitively missing, as that’s treated as a correctness bug, not just a tuning problem.[[MySQL missing rows](https://github.com/airbytehq/airbyte/issues/53614)] The knowledge sources do not show any supported way today to override fetch_size manually.Bogdan
11/26/2025, 12:23 PMkapa.ai
11/26/2025, 12:24 PMkapa.ai
11/26/2025, 12:24 PM_airbyte_emitted_at, timestamps in Avro logical types, etc.) are stored in UTC. For JSON→Avro conversion (used by blob destinations like S3), date-time fields become Avro timestamp-micros representing “the number of microseconds from the unix epoch, 1 January 1970 000000.000000 UTC”, and the value is stored in UTC regardless of server timezone.[[JSON→Avro rules](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules)] 2. Airbyte temporal types are defined independently of server timezone Airbyte defines five temporal types: date, timestamp_with_timezone, timestamp_without_timezone, time_with_timezone, time_without_timezone. These are represented as RFC3339 strings with format: date or format: date-time and disambiguated via airbyte_type.[[Data types – specific](https://docs.airbyte.com/platform/understanding-airbyte/supported-data-types#specific-types)] Destinations are expected to honor whether a value is “with timezone” or “without timezone”, and to default to the with‑timezone variant if it’s unspecified.[[Data types – specific](https://docs.airbyte.com/platform/understanding-airbyte/supported-data-types#specific-types)] 3. JSON→Avro / Parquet conversions normalize to UTC For blob destinations, built‑in JSON date-time formats are mapped to timestamp-micros and “logical type fields will be stored as UTC, respecting timezone as applicable.”[[JSON→Avro rules](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules)] For S3 migration, timestamps with timezone are explicitly converted to UTC; old mixed string+logical unions were removed, but the data semantics remain UTC-based.[[S3 migration – date/time](https://docs.airbyte.com/integrations/destinations/s3-migrations#date-and-time-types)] Implication for your case (source in UTC, server in Europe/Moscow): - If the source sends proper UTC timestamps (e.g. 2024-01-01T12:00:00Z or +00:00), Airbyte treats them as UTC and stores UTC instants. The server’s Europe/Moscow timezone does not cause a shift in those stored values. - Any “shift” you see will usually come from how your destination or BI tool interprets/prints those UTC instants (e.g., automatically displaying them in local time), not from Airbyte changing the underlying moment in time. The provided sources do not describe any behavior where Airbyte re‑bases timestamps based solely on the host/server timezone, so based on current docs, you should not expect an automatic time shift just because the Airbyte server runs in Europe/Moscow.Bogdan
11/26/2025, 1:35 PMkapa.ai
11/26/2025, 1:35 PMkapa.ai
11/26/2025, 1:35 PMReplicatedMergeTree / replicated tables for you, nor to customize the exact CREATE TABLE statement in a way needed for multi‑node replication (for example, adding ON CLUSTER or a ReplicatedMergeTree engine). A forum post confirms that “today the connector only works with single instance” and that achieving cluster‑wide replicated behavior would require a customized connector implementation.[[Destination to Clickhouse cluster](https://discuss.airbyte.io/t/1026)] Regarding creating the table yourself: - The legacy (v1, archived) ClickHouse destination explicitly creates its own tables with a fixed schema (_airbyte_ab_id, _airbyte_emitted_at, _airbyte_data).[[ClickHouse archived](https://docs.airbyte.com/integrations/destinations/clickhouse-deprecated)] - For the new v2 ClickHouse destination, the docs state that it “writes each stream to its own table” and creates tables in the configured/default database or in the database corresponding to the stream namespace, with typed columns according to the mapping.[[ClickHouse output schema](https://docs.airbyte.com/integrations/destinations/clickhouse#output-schema)] However, the provided sources do not clearly state whether v2 will: - Reuse an existing table if the schema matches, or - Attempt to CREATE / ALTER and thus fail or modify your pre‑created replicated table. One GitHub issue for the new ClickHouse destination shows that Airbyte does attempt to ALTER tables it created (e.g., changing primary‑key column nullability), and there is currently no configuration to tell it “don’t change destination columns.”[[Clickhouse fails second incremental sync](https://github.com/airbytehq/airbyte/issues/65915)] This strongly suggests that if you pre‑create a replicated table with the same name, Airbyte may still try to manage its schema and could conflict with your engine/cluster settings. So, based on the available information: - Airbyte cannot natively create replicated ClickHouse tables for you. - Pre‑creating a replicated table with the same name is not guaranteed to be safe: the connector may still attempt `CREATE`/`ALTER` operations, and there is no documented option to fully disable that behavior.[[Clickhouse fails second incremental sync](https://github.com/airbytehq/airbyte/issues/65915)] If you need strict control over replicated tables, the current docs and discussions point to building or forking a custom ClickHouse destination that uses your desired CREATE TABLE / engine definitions.[[Destination to Clickhouse cluster](https://discuss.airbyte.io/t/1026)]Bogdan
11/27/2025, 8:43 AMkapa.ai
11/27/2025, 8:43 AMkapa.ai
11/27/2025, 8:44 AMupdated_at).[[Inc. append+dedup overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped); [Rules examples](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#rules)] For the ClickHouse destination v2, this mode is explicitly supported, and deduplication leverages ReplacingMergeTree on the ClickHouse side.[[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] ### Impact on load Based on the docs: - On Airbyte: - Versus plain Incremental Append, Incremental Append + Deduped adds the work of computing a deduplicated “final” table from the history data.[[Inc. append+dedup overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] - The docs do not quantify the extra CPU/IO, but logically it is more work than Append alone. The knowledge sources don’t provide precise performance numbers. - On ClickHouse: - The connector uses ReplacingMergeTree to implement deduplication, so ClickHouse does more merge/dedup work in the background than with pure append tables.[[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] - The docs only advise on choosing an appropriate cursor type for good deduplication (integer, date, or timestamp); otherwise it falls back to _airbyte_extracted_at for ordering, which can affect correctness or performance of merges, but they do not provide explicit load benchmarks.[[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] So: - Yes, it inherently adds some extra load compared to simple Incremental Append, on both Airbyte and ClickHouse, due to history storage and dedup/merge work. - The exact magnitude of that load is not specified in the available documentation.Bogdan
11/27/2025, 8:47 AMkapa.ai
11/27/2025, 8:47 AMkapa.ai
11/27/2025, 8:47 AMReplacingMergeTree for compaction‑style deduplication” in Incremental Append + Deduped mode, but they do not spell out the exact timing (during the Airbyte sync workflow vs. entirely deferred to ClickHouse merges). [[ClickHouse v2 beta](https://github.com/airbytehq/airbyte/discussions/62035); [ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] From this we can say confidently: - Airbyte configures tables so that deduplication is handled using ReplacingMergeTree on the ClickHouse side. [[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] - The documentation does not clearly describe whether Airbyte also performs additional dedup logic during the sync itself, or relies solely on ClickHouse’s background merge/compaction process. So, based on the available sources, the mechanism involves ReplacingMergeTree in ClickHouse, but the exact division of work between “during sync” vs. “afterward by the engine” is not explicitly documented.Bogdan
11/27/2025, 9:34 AMkapa.ai
11/27/2025, 9:34 AMkapa.ai
11/27/2025, 9:35 AMconfirmed_lsn / confirmed_flush_lsn). When that happens, Airbyte can’t continue reading changes incrementally and fails the sync with: > Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency… [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs); [AlloyDB issue](https://github.com/airbytehq/airbyte/issues/48744)] From the docs, this generally occurs because: 1. Required WAL segments were removed - Heavy write activity caused Postgres to purge/archive WAL files needed to continue from Airbyte’s last saved LSN. - This is controlled mainly by wal_keep_size (and, on managed services, similar WAL retention settings). [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs); [CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#general-airbyte-configuration)] 2. Syncs are too infrequent for the WAL retention - If your sync interval is longer than the effective WAL retention window, WAL segments will be discarded before Airbyte can read them. [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#general-airbyte-configuration)] 3. Rare case: destination failed after WAL was acknowledged - Source successfully read and acknowledged WAL, but destination failed; on retry, the WAL segment is no longer available because Postgres already considered it consumed. [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs)] In some recent issues, this also shows up after configuration changes or state inconsistencies (e.g., switching tables, removing schemas, or connector behavior around confirmed_flush_lsn vs. saved LSN). [[Issue 66091](https://github.com/airbytehq/airbyte/issues/66091); [Issue 55846](https://github.com/airbytehq/airbyte/issues/55846); [Issue 49802](https://github.com/airbytehq/airbyte/issues/49802)] ### What you can do 1. Reset or recreate the connection (required to recover) - With Invalid CDC position behavior set to Re-sync data, Airbyte will trigger a sync from scratch when it detects this mismatch. [[Issue 49802](https://github.com/airbytehq/airbyte/issues/49802)] - Otherwise you may need to Reset connection in the UI, or, in some edge cases (like stale state on removed tables), even delete and recreate the connection, as one user reported. [[Issue 66091](https://github.com/airbytehq/airbyte/issues/66091)] 2. Increase WAL retention / keep WAL longer - Raise wal_keep_size (or the managed‑service equivalent like wal_keep_size / max_slot_wal_keep_size on AlloyDB). [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs); [AlloyDB issue](https://github.com/airbytehq/airbyte/issues/48744)] - Be aware this increases disk usage. 3. **Increase sync frequency so it’s shorter than WAL retention** - Docs recommend that sync frequency must be less than the WAL retention period (e.g., if retention is 3 days, sync at least every 2 days). [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#general-airbyte-configuration)] - For most workloads, daily or every few hours is suggested; for low‑activity DBs, just ensure the interval is well within retention. [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#general-airbyte-configuration)] 4. If you have low write volume and LSN doesn’t advance well - In some cases, WAL disk consumption and LSN behavior with low transaction volumes can be tricky. Docs suggest adding a high‑volume or heartbeat table to the publication and, optionally, using a Debezium heartbeat query to keep the LSN progressing. [[Advanced WAL troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] 5. If it keeps happening even with high WAL settings and frequent syncs - This may be related to the open bugs around Saved offset is before replication slot's confirmed lsn (e.g., mismatch between saved state LSN and confirmed_flush_lsn, or behavior on managed services like AlloyDB). [[Issue 48744](https://github.com/airbytehq/airbyte/issues/48744); [Issue 55846](https://github.com/airbytehq/airbyte/issues/55846)] - In that case, collecting logs that show the saved LSN vs pg_replication_slots.confirmed_flush_lsn and attaching them to the relevant GitHub issue would be the recommended next step. If you share: - Your Postgres flavor (self‑managed, RDS, AlloyDB, etc.) - Current wal_keep_size / retention settings - Sync frequency - Whether this happens right after the first (snapshot) sync or only after later syncs I can narrow down which of the documented causes is most likely in your setup and which of the above mitigations should be prioritized.Bogdan
11/27/2025, 12:36 PMkapa.ai
11/27/2025, 12:36 PMkapa.ai
11/27/2025, 12:36 PMairbyte-worker and logging system - Users have reported that airbyte-worker memory keeps increasing over days or weeks, even with a steady or idle workload, sometimes reaching many GB and thousands of threads until the worker must be restarted.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309); [Gradual increase in memory](https://discuss.airbyte.io/t/2249)] - The team acknowledged a suspected memory leak in the logging system, and suggested restarting airbyte-server and airbyte-worker as a workaround while they investigate.[\[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] - In Kubernetes deployments, users have seen the worker pod’s memory steadily grow from a few hundred MB up to its limit, at which point it gets OOM‑killed; restarting the deployment resets usage temporarily.[\[Memory leak worker pod](https://github.com/airbytehq/airbyte/issues/56944)] 2. Server process consuming large amounts of RAM over time - There are reports of airbyte-server using tens of GB of RAM (e.g., 48–50 GB on a 64 GB host) with only a few short‑lived jobs, and memory not being released even when no jobs are running.[\[Unusual RAM usage server](https://discuss.airbyte.io/t/2167)] - Restarting the server temporarily reduces usage, but it grows again when jobs run. The root cause is not fully documented in the provided sources, but it is treated as a bug and tracked on GitHub.[\[Unusual RAM usage server](https://discuss.airbyte.io/t/2167)] 3. Worker memory slowly increasing between syncs - Another user observed that even with a single scheduled sync every few hours, idle memory usage for airbyte-worker and other containers kept increasing after each sync and never went back down.[\[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083)] - Support suggested that the worker is doing internal tasks even when no syncs are running, and recommended adding resource constraints as a safeguard (accepting possible Java OOM inside the container instead of exhausting the host), and again pointed to a suspected memory leak.[\[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083); \[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] 4. Thread / logging‑related leaks - One detailed investigation found that the worker had ~30k threads and that most of them came from the log4j S3 appender, with threads blocking on locks and logs not flushing properly to S3; this was characterized as a resource leak in the worker.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309)] - Another forum thread mentions that the team is investigating a memory‑leaky logging system, again aligning with what you’re seeing.[\[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] Given these references, the most likely causes for “RAM keeps increasing even with syncs turned off” are: - A known memory/resource leak in airbyte-worker and/or airbyte-server, often related to logging and background threads, not strictly tied to active syncs.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309); \[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] - Java processes not releasing heap back to the OS even when they are logically idle; combined with leaks, this looks like ever‑growing RAM usage.[\[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083)] ### What you can do right now Based on the documented workarounds: 1. Restart airbyte-server and airbyte-worker periodically - This is the explicit workaround suggested while the memory‑leak issue in the logging system is under investigation.[\[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] - In Kubernetes, some users set memory limits so that hitting the limit causes a pod restart, combined with the container‑orchestrator feature to avoid impacting active syncs.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309)] 2. Add resource limits to isolate the problem - For Docker deployments, set memory limits on airbyte-worker (and possibly airbyte-server) so they cannot exhaust the host; if limits are too strict you may see Java OOM errors, but you’ll protect the rest of the system.[\[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083)] - For Kubernetes, set reasonable resources.requests/limits on the worker pod; users have observed leaks even with limits, but at least the pod will be restarted when it hits the cap.[\[Memory leak worker pod](https://github.com/airbytehq/airbyte/issues/56944)] 3. Monitor which component is growing - Use docker stats or kubectl top pods / container metrics to confirm whether it’s specifically airbyte-server, airbyte-worker, or another pod that’s increasing over time.[\[Gradual increase in memory](https://discuss.airbyte.io/t/2249); \[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083)] - This will help you match your situation to the known issues (e.g., worker vs server vs logging to S3). 4. Check for S3 logging + very high thread counts (if applicable) - If you configured log shipping to S3 and see very high thread counts in the worker, you may be hitting the log4j S3 appender issue.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309)] - In that case, reconsider S3 logging config or contact the team with a thread dump, as that user did. Because the provided sources don’t include a final, merged fix for these leaks, I can’t point to a specific version where it’s definitively resolved. If you haven’t yet: - Check your Airbyte version against the issues above, and - Consider opening or updating a GitHub issue with your version, deployment type, and which container is leaking, referencing those existing reports. That will both help you get guidance on whether a fix exists in a newer version, and give the team more data on how often this still occurs.Bogdan
11/27/2025, 1:18 PMkapa.ai
11/27/2025, 1:18 PMkapa.ai
11/27/2025, 1:19 PMabctl is trying to pull images that don’t exist on Docker Hub for 2.0.1, notably: - airbyte/async-profiler:2.0.1 - airbyte/utils:2.0.1 Very similar issues have been reported: - abctl local / local setup hanging or failing on airbyte/async-profiler:1.6.2 or 1.7.2 because those tags don’t exist. [[Local setup issue](https://github.com/airbytehq/airbyte/issues/61339)] - Helm charts 2.0.0 / 2.0.1 were pulled from the repo because they referenced non‑existing images (e.g., airbyte/webapp:2.0.0), and users were instructed to deploy Airbyte 2.0.1 using Helm chart 1.9.1 instead. [[Chart 2.0.0 problems](https://github.com/airbytehq/airbyte/issues/68134); [Use chart 1.9.1](https://github.com/airbytehq/airbyte/issues/68136)] In short: you’ve hit a bad image/tag combination; abctl is asking Docker to pull tags that aren’t published. Given the available information, the practical options are: 1. Use a supported chart / version combination If you’re deploying via Helm, follow the guidance to use Helm chart v1.9.1 to deploy Airbyte v2.0.1, not the removed 2.0.x charts that reference missing images.[[Use chart 1.9.1](https://github.com/airbytehq/airbyte/issues/68136); [webapp:2.0.0 not found](https://github.com/airbytehq/airbyte/issues/68134)] 2. Avoid image tags that don’t exist The async-profiler tag problem is known: trying to pull airbyte/async-profiler:1.6.2 or 1.7.2 fails with manifest unknown because those tags are not present on Docker Hub.[[Local setup issue](https://github.com/airbytehq/airbyte/issues/61339)] Your logs show the same pattern for 2.0.1. The knowledge sources do not document a way to override these image names in abctl local today, so there is no supported configuration knob to “skip” or change those images in that workflow. 3. If you’re using a local abctl install: - The docs show `abctl`’s images manifest being used to list images and then tag/push them to a custom registry, but they assume the images actually exist on Docker Hub for that version. [[Custom image registry](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#configure-a-custom-image-registry)] - When the manifest includes non‑existent tags (as with the async-profiler example), docker pull simply fails and the install hangs or errors, as in the open GitHub issue.[[Local setup issue](https://github.com/airbytehq/airbyte/issues/61339)] Because the provided sources don’t include a fix for airbyte/async-profiler:2.0.1 or airbyte/utils:2.0.1, they don’t show a way to complete an abctl local install that depends on those tags. The documented resolutions in similar cases are: - Switch to a version / chart combination whose images all exist (e.g., Airbyte 1.8 + --low-resource-mode resolved the async‑profiler issue for one user). [[Local setup issue](https://github.com/airbytehq/airbyte/issues/61339)] - Update the Helm repo and install with a chart that has correct image references (Helm chart 1.9.1 for Airbyte 2.0.1). [[Use chart 1.9.1](https://github.com/airbytehq/airbyte/issues/68136)] Given the limitations of the current knowledge sources, I can’t point to a specific abctl flag or patch that fixes the 2.0.1 `async-profiler`/`utils` tags; the safest documented path is to align your deployment with a version / chart combination that does not reference missing images.Bogdan
11/28/2025, 7:52 AMALTER SUBSCRIPTION <subscription_name> REFRESH PUBLICATION; on the subscriber side, and is it possible to perform this action through Airbyte? @kapa.aikapa.ai
11/28/2025, 7:52 AMkapa.ai
11/28/2025, 7:53 AMALTER PUBLICATION on the source DB, for example: sql ALTER PUBLICATION <publication_name> ADD TABLE tbl_new1, tbl_new2; This is the same pattern shown for the initial setup (where you create the publication with specific tables).[[Cloud SQL CDC step 5](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#step-5-create-publication-and-replication-identities-for-each-postgres-table)] So the general procedure on the source: 1. Ensure each new table has an appropriate replication identity: sql ALTER TABLE tbl_new1 REPLICA IDENTITY DEFAULT; or REPLICA IDENTITY FULL if needed.[[Cloud SQL CDC step 5](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#step-5-create-publication-and-replication-identities-for-each-postgres-table)] 2. Add the new table(s) to the existing publication: sql ALTER PUBLICATION <publication_name> ADD TABLE tbl_new1, tbl_new2; ```[[Cloud SQL CDC step 5](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#step-5-create-publication-and-replication-identities-for-each-postgres-table)] Then, in **Airbyte**: - Manually **refresh the source schema** on the connection’s *Schema* tab and enable the new stream(s).[[Schema refresh](https://docs.airbyte.com/platform/using-airbyte/schema-change-management#manually-refresh-the-source-schema)] - For CDC, new tables/columns must be snapshotted once before CDC can track changes; use *Refresh Stream* for the new table to trigger that snapshot.[[CDC limitations – adding schemas/columns](https://docs.airbyte.com/platform/understanding-airbyte/cdc#limitations)] ### 2. `ALTER SUBSCRIPTION ... REFRESH PUBLICATION` The knowledge sources do **not** mention PostgreSQL subscriptions or the command: ```sql ALTER SUBSCRIPTION <subscription_name> REFRESH PUBLICATION; Airbyte’s CDC implementation talks only about publications and how Airbyte reads from them; it does not configure or manage any Postgres subscriptions.[[Cloud SQL CDC step 5](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#step-5-create-publication-and-replication-identities-for-each-postgres-table); [CDC overview](https://docs.airbyte.com/platform/understanding-airbyte/cdc)] Based on the available docs: - Whether you need ALTER SUBSCRIPTION ... REFRESH PUBLICATION is not addressed; Airbyte itself does not act as a logical replication subscriber. - There is no documented way in Airbyte to issue that command on your behalf; you’d run any subscription maintenance directly in Postgres, outside of Airbyte. So Airbyte can help by: - Requiring and using a correctly configured publication. - Letting you refresh schema and add new tables/columns inside the Airbyte connection. But it does not manage PostgreSQL subscriptions, and the provided documentation does not describe integrating ALTER SUBSCRIPTION ... REFRESH PUBLICATION into Airbyte.Bogdan
12/03/2025, 8:06 AMkapa.ai
12/03/2025, 8:06 AMkapa.ai
12/03/2025, 8:07 AMpublic_booking_prices), this is expected if the snapshot can’t finish within 8 hours. The docs recommend: - For large / very large databases, increase Initial Load Timeout to 12–24 hours, up to the maximum of 24 hours, based on observed snapshot duration.[[CDC source config](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] So in your case: - The log is not an error in CDC itself, but a timeout safeguard. - If you want the initial snapshot of public_booking_prices to complete in a single run, increase Initial Load Timeout in the source’s Advanced CDC settings, then rerun. - If you leave it at 8h, expect the snapshot to progress across multiple attempts while CDC keeps changes up to date.Bogdan
12/03/2025, 8:09 AMkapa.ai
12/03/2025, 8:09 AMkapa.ai
12/03/2025, 8:09 AM8 to 12 and save. [[Db2 CDC config](https://docs.airbyte.com/integrations/enterprise-connectors/source-db2-enterprise#configuring-cdc-in-airbyte); [CDC source config](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] If your specific source connector UI does not expose an Initial Load Timeout parameter, the knowledge sources don’t document another supported way to push the snapshot timeout to 12 hours for that connector.Bogdan
12/03/2025, 8:15 AMkapa.ai
12/03/2025, 8:15 AMkapa.ai
12/03/2025, 8:15 AMInitial Waiting Time is not a global Helm/CheLM setting; it’s a per‑connection, per‑source option exposed in the connector configuration UI for CDC‑capable sources (for example, Postgres, Db2 Enterprise, MongoDB v2, SAP HANA, etc.). From the docs: - For Postgres, “Initial Waiting Time in Seconds (Advanced)” is an advanced CDC parameter that controls how long the connector waits for WAL / CDC events before timing out.[[Postgres CDC troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-setting-up-initial-cdc-waiting-time); [Postgres issue](https://github.com/airbytehq/airbyte/issues/50435)] - CDC best‑practices describe Initial Waiting Time as a source configuration setting that affects both the snapshot and CDC incremental phases; the range “varies by source (check your source configuration page).”[[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices); [Next CDC best practices](https://docs.airbyte.com/platform/next/understanding-airbyte/cdc-best-practices)] You can see similar fields in other CDC‑style sources: - MongoDB v2 lists initial_waiting_seconds as “Initial Waiting Time in Seconds (Advanced)” in its config fields.[[MongoDB v2 reference](https://docs.airbyte.com/integrations/sources/mongodb-v2#reference)] - Db2 Enterprise and SAP HANA CDC expose related advanced timing options (e.g., Initial Load Timeout, Initial Load Timeout in Hours, etc.) in the connector configuration, not in Helm values.[[Db2 CDC config](https://docs.airbyte.com/integrations/enterprise-connectors/source-db2-enterprise#configuring-cdc-in-airbyte); [SAP HANA CDC](https://docs.airbyte.com/integrations/enterprise-connectors/source-sap-hana#change-data-capture-cdc)] So, to change Initial Waiting Time: 1. Open your source in the Airbyte UI. 2. Click Edit configuration. 3. Scroll to the Advanced / CDC section. 4. Look for “Initial Waiting Time in Seconds (Advanced)” (or similarly named) and adjust it within the allowed range (for Postgres: 120–1200 seconds).[[Postgres CDC troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-setting-up-initial-cdc-waiting-time)] The provided knowledge sources do not show any values.yaml (Helm) key for Initial Waiting Time, so there’s no documented way to set it globally via Helm; it’s configured per connector in the UI.Bogdan
12/03/2025, 8:47 AMkapa.ai
12/03/2025, 8:47 AMkapa.ai
12/03/2025, 8:48 AMTEMPORAL_HISTORY_RETENTION_IN_DAYS “defines the retention period of the job history in Temporal” and defaults to 30 days.[\[Configuring Airbyte 1.8](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-airbyte#data-retention); [Configuring Airbyte (next)](https://docs.airbyte.com/platform/next/operator-guides/configuring-airbyte#data-retention)] - Forum answers explicitly recommend changing this variable to reduce storage usage and help clean Temporal history/log-related data (for both Docker and K8s deployments).[\[Forum 2200](https://discuss.airbyte.io/t/2200); [Forum 2417](https://discuss.airbyte.io/t/2417); [GitHub discussion 55233](https://github.com/airbytehq/airbyte/discussions/55233); [Issue 55812](https://github.com/airbytehq/airbyte/issues/55812)] So today: - Default retention window: 30 days of Temporal job history. - You can adjust it by setting TEMPORAL_HISTORY_RETENTION_IN_DAYS to the number of days you want. ### 2. Older / legacy workspace log sweeping Older scheduler‑based deployments used: - MINIMUM_WORKSPACE_RETENTION_DAYS (default 1 day) - MAXIMUM_WORKSPACE_RETENTION_DAYS (default 60 days) - MAXIMUM_WORKSPACE_SIZE_MB (default 5 GB) to control deletion of sync logs in the workspace.[\[Rotation logs](https://discuss.airbyte.io/t/332); [Log sweeping](https://discuss.airbyte.io/t/2714)] However: - That mechanism was removed with the old Scheduler, and the documented replacement is again TEMPORAL_HISTORY_RETENTION_IN_DAYS.[\[Log sweeping](https://discuss.airbyte.io/t/2714); [Forum 2200](https://discuss.airbyte.io/t/2200); [Forum 2417](https://discuss.airbyte.io/t/2417)] ### 3. Practical answer to your questions - How long are logs kept by default? Airbyte keeps job history (and associated Temporal log history) for 30 days by default via TEMPORAL_HISTORY_RETENTION_IN_DAYS.[\[Configuring Airbyte 1.8](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-airbyte#data-retention)] - Is there a size limit? Older scheduler‑based setups used a 5 GB workspace size limit (MAXIMUM_WORKSPACE_SIZE_MB default 5 GB) as part of log sweeping,[\[Rotation logs](https://discuss.airbyte.io/t/332)] but this config was removed and the current docs do not describe an equivalent size‑based limit in the new system. The knowledge sources do not clearly state a current global size cap beyond that legacy behavior. - Can you adjust these values? - Yes: you can adjust retention in days by setting TEMPORAL_HISTORY_RETENTION_IN_DAYS (for example, to 7 or 1 day).[\[Configuring Airbyte](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#data-retention); [Issue 55812](https://github.com/airbytehq/airbyte/issues/55812)] - The older MINIMUM_WORKSPACE_RETENTION_DAYS, MAXIMUM_WORKSPACE_RETENTION_DAYS, and MAXIMUM_WORKSPACE_SIZE_MB are no longer effective in current deployments using Temporal.[\[Log sweeping](https://discuss.airbyte.io/t/2714)] If you need stricter disk control (e.g., object storage for logs filling up), the sources also show operators manually deleting old log objects (e.g., in MinIO) in addition to tuning TEMPORAL_HISTORY_RETENTION_IN_DAYS.[\[Issue 55812](https://github.com/airbytehq/airbyte/issues/55812)]Bogdan
12/03/2025, 10:01 AMkapa.ai
12/03/2025, 10:01 AMkapa.ai
12/03/2025, 10:02 AMNetworkError when attempting to fetch resource message fits the pattern of the browser trying to call an old or invalid endpoint. 2. Frontend calling a wrong / non‑existent endpoint For example, in 1.7.1 the Connector Builder UI called /api/v1/connector_builder/manifest/resolve, which didn’t exist in that version and always returned 404, breaking only that feature.[[Connector Builder 404](https://github.com/airbytehq/airbyte/issues/63770)] In that case, the backend was fine; the bug was isolated to one UI path. 3. Local networking / DNS / proxy issues specific to one machine Other forum threads about “Cannot reach server” or failing connection tests point to DNS / firewall / proxy issues when only some calls from the browser fail while the Airbyte instance itself is healthy.[[Google Sheets DNS](https://discuss.airbyte.io/t/2626); [GKE cannot reach server](https://discuss.airbyte.io/t/763)] Given the available information: - The Airbyte backend is likely OK (syncs run). - The issue is likely your browser or local network environment. What to try (in order): 1. Hard‑refresh / clear cache and storage for the Airbyte URL - Fully clear site data (cache, local storage, indexedDB) for your Airbyte URL and reload, as done in the 1.6.0 issue.[[New connection broken](https://github.com/airbytehq/airbyte/issues/58110)] 2. Try a different browser or incognito window on your machine - If that works, it confirms a cache/extension issue in your main browser. 3. Check the browser DevTools → Network tab - Look at the failing request when you open the “new connection” page: status code, URL. This is what support asked for in similar “cannot reach server / failed schema fetch” issues.[[GitHub schema fetch](https://discuss.airbyte.io/t/905); [GKE cannot reach server](https://discuss.airbyte.io/t/763)] 4. Compare with a teammate’s machine - If they can create connections, have them open DevTools and compare the working request URL and response to your failing one. The knowledge sources don’t show a server‑side Airbyte bug that matches “only one user gets NetworkError on new connection while syncs still work,” so the evidence points strongly to local browser cache or network configuration on your machine.Bogdan
12/04/2025, 1:01 PMkapa.ai
12/04/2025, 1:01 PMkapa.ai
12/04/2025, 1:01 PMdate, timestamp_with_timezone, timestamp_without_timezone, time_with_timezone, time_without_timezone, all represented as RFC 3339 strings plus an airbyte_type annotation to specify whether the value includes a timezone.[[Data types](https://docs.airbyte.com/platform/understanding-airbyte/supported-data-types#specific-types)] - For blob destinations (Avro/Parquet), JSON date-time values are converted to Avro timestamp-micros and are stored as UTC, “respecting timezone as applicable.”[[JSON–Avro](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules)] - Some destinations (for example, GCS Data Lake / Iceberg) explicitly convert “time with timezone” and “timestamp with timezone” to UTC before writing, by design.[[GCS Data Lake schema](https://docs.airbyte.com/integrations/destinations/gcs-data-lake#output-schema)] So: - Airbyte generally normalizes time‑with‑timezone data to UTC at the storage level. - There is no documented setting to automatically convert all timestamps into an arbitrary local timezone (e.g., Europe/Moscow) during sync. - If you need local time, you typically handle the conversion in the destination system or downstream queries/BI tools, not in Airbyte itself. The knowledge sources do not describe any feature that globally “converts Airbyte to local time,” so any such behavior would need to be implemented outside of Airbyte or via custom connector logic.Bogdan
12/05/2025, 10:19 AMmax_sync_workers configuration is currently set to the default." @kapa.aikapa.ai
12/05/2025, 10:20 AMkapa.ai
12/05/2025, 10:20 AMMAX_SYNC_WORKERS sync jobs at once. With abctl (Kubernetes under the hood), this is set via env or Helm values: yaml worker: maxSyncWorkers: 10 # example maxCheckWorkers: 5 or, in env form: yaml MAX_SYNC_WORKERS=10 MAX_CHECK_WORKERS=5 Defaults are 10 sync workers and 5 check workers per worker container.[[Config worker](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker)] With 3 worker replicas and default MAX_SYNC_WORKERS=10, you can run up to 30 syncs in parallel in theory, but other limits may be capping you. 2. Global concurrency / scheduler limits Older docs and forum answers mention: - SUBMITTER_NUM_THREADS – hard upper limit on total concurrent jobs across the system (deprecated with the new Temporal scheduler, but still referenced in some setups).[[Scaling k8s parallelism](https://discuss.airbyte.io/t/826); [How to configure concurrency](https://discuss.airbyte.io/t/1077)] - With the new scheduler, concurrency is effectively: parallel syncs ≈ number of worker pods × MAX_SYNC_WORKERS (also bounded by available ports).[[How to configure concurrency](https://discuss.airbyte.io/t/1077)] If SUBMITTER_NUM_THREADS is still low in your environment, it can serialize syncs even if you have multiple workers and higher MAX_SYNC_WORKERS.[[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)] 3. Temporal worker ports Each worker pod needs enough ports to talk to job pods: - TEMPORAL_WORKER_PORTS defines the local ports the worker uses; Helm exposes 9001–9040 by default (40 ports).[[Config worker](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker)] - In practice, this caps how many jobs a single worker can run; forum guidance suggests something like: parallel syncs per worker ≈ min(MAX_SYNC_WORKERS, TEMPORAL_WORKER_PORTS/4).[[How to configure concurrency](https://discuss.airbyte.io/t/1077); [Scaling k8s parallelism](https://discuss.airbyte.io/t/826)] If TEMPORAL_WORKER_PORTS is too small, you won’t get the parallelism implied by MAX_SYNC_WORKERS. 4. Cluster resources If your Kubernetes cluster is under‑provisioned (CPU/memory), pods for additional syncs may stay pending, so you effectively see only one running at a time even though Airbyte wants to run more.[[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)] --- ### What to change / check in your case Given you deployed with abctl and have 3 worker replicas but only 1 sync at a time: 1. Increase MAX_SYNC_WORKERS on workers Set it explicitly higher than 1 (e.g., 5 or 10) via the worker config (Helm values or env), then restart/redeploy workers so the new value is applied.[[Config worker](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker)] 2. Ensure global limits aren’t 1 If your environment still uses SUBMITTER_NUM_THREADS, make sure it’s not set to 1; forum examples for ~100 concurrent syncs use values like: text SUBMITTER_NUM_THREADS = 200 MAX_SYNC_WORKERS = 100 Number of workers = 3 [[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)] 3. Check TEMPORAL_WORKER_PORTS Confirm it exposes enough ports (e.g., 9001–9040 or more). If you’ve customized it to a very small range, that can cap concurrency per worker.[[Config worker](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker); [How to configure concurrency](https://discuss.airbyte.io/t/1077)] 4. Verify cluster capacity Use kubectl get pods / metrics to see if additional sync pods (orchestrator, read, write) are pending due to lack of resources. If so, you’ll need more node capacity or lower per‑job resource requests.[[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)] Because abctl specifics (exact flags/fields) are not detailed in the provided sources, I can’t give you the exact file/command to edit, but the knobs you need to adjust are: - worker.replicaCount (you already have 3) - MAX_SYNC_WORKERS / worker.maxSyncWorkers - Possibly SUBMITTER_NUM_THREADS (if present) - TEMPORAL_WORKER_PORTS - Underlying K8s resources If you share how you’re configuring Airbyte with abctl (values file vs defaults), you can map these directly into that config.Bogdan
12/05/2025, 10:32 AMkapa.ai
12/05/2025, 10:32 AMkapa.ai
12/05/2025, 10:33 AMmaxSyncWorkers is only one of the limits You correctly set: yaml worker: maxSyncWorkers: 10 replicaCount: 3 So in theory you can run up to 30 syncs in parallel. But concurrency is also constrained by: - The number of worker pods (replicaCount) - The per‑worker max (maxSyncWorkers) - Temporal worker ports (TEMPORAL_WORKER_PORTS) - Global scheduler behavior and available cluster resources [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)]. 2. You haven’t changed TEMPORAL_WORKER_PORTS In Kubernetes deployments, each worker can only run as many jobs as it has Temporal worker ports. The docs and forum guidance describe that concurrency per worker is effectively: > parallel syncs per worker ≈ min(MAX_SYNC_WORKERS, TEMPORAL_WORKER_PORTS / 4) [[Scaling k8s parallelism](https://discuss.airbyte.io/t/826); [How to configure concurrency](https://discuss.airbyte.io/t/1077)]. If TEMPORAL_WORKER_PORTS is left at a very small default or misconfigured, it can cap you to 1 sync per worker even though maxSyncWorkers is 10. The docs don’t show the exact default in your abctl setup, so this is an area of uncertainty, but it is explicitly called out as affecting concurrency [[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)]. 3. Cluster resources can still serialize jobs Even with your resource limits, if the Kubernetes cluster doesn’t have enough free CPU/memory to schedule the orchestrator, read, and write pods for multiple syncs, you’ll effectively see only one running at a time. The scaling docs note that each concurrent sync needs at least 3 extra pods, and under‑provisioned clusters often manifest as “stuck” or effectively serialized jobs [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)]. 4. Other concurrency knobs are at defaults The knowledge sources mention older/global limits like SUBMITTER_NUM_THREADS in some Kubernetes setups, which can cap total concurrent jobs regardless of MAX_SYNC_WORKERS [[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)]. However, they don’t show how (or if) this is exposed in your abctl‑based deployment, so I can’t say for sure whether it’s affecting you. --- ### What you can try next (based on the docs) 1. Check / increase TEMPORAL_WORKER_PORTS Ensure it covers enough ports per worker so that TEMPORAL_WORKER_PORTS / 4 is at least 10, to match your maxSyncWorkers target [[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)]. 2. Verify that multiple sync pods are actually being scheduled Use kubectl get pods and look for multiple orchestrator-, read-, and write- pods in Running state at the same time. If they’re Pending, you likely need more cluster capacity or lower per‑job resource defaults [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)]. 3. Confirm that your maxSyncWorkers is applied Make sure the running worker pods actually have MAX_SYNC_WORKERS=10 in their environment (e.g., via kubectl describe pod). The scaling docs assume this env is what drives concurrency [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits)]. The knowledge sources don’t provide abctl‑specific YAML examples beyond the generic Helm values, so I can’t give an exact abctl command, but the main missing piece from what you shared is likely the Temporal worker ports and/or actual cluster capacity, not the maxSyncWorkers setting itself.