I have set `max_check_workers=10` and `max_sync_wo...
# ask-ai
b
I have set
max_check_workers=10
and
max_sync_workers=10
in my Airbyte setup, but syncs are still running sequentially instead of in parallel. Has anyone encountered this issue before? What could be causing this behavior despite having these values set? Any suggestions would be greatly appreciated. @kapa.ai
k
A few things in the docs and forum threads could explain why you still see sequential syncs even with
max_check_workers=10
and `max_sync_workers=10`: 1. Worker limits are per worker pod, not global
MAX_SYNC_WORKERS
and
MAX_CHECK_WORKERS
(or the equivalent Helm
worker.maxSyncWorkers
,
worker.maxCheckWorkers
) are per Airbyte worker container limits, not a guarantee of global concurrency.[\[Configuring worker\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker) If you only have one worker pod and other constraints are low, you may still see just a few (or even one) sync at a time. 2. Overall submitter limit (
SUBMITTER_NUM_THREADS
)
In K8s deployments,
SUBMITTER_NUM_THREADS
is a hard upper bound on total concurrent jobs in the system, regardless of
MAX_*_WORKERS
.[\[Scaling k8s parallelism\]](https://discuss.airbyte.io/t/826) If this is still at a low default, it can effectively serialize your syncs even though
MAX_SYNC_WORKERS
is 10. 3. Temporal worker ports can cap parallelism Each worker pod can only run as many jobs as it has Temporal worker ports exposed. The Helm chart exposes ports 9001–9040 by default, i.e., at most 40 parallel jobs per worker.[\[Configuring worker\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker) If
TEMPORAL_WORKER_PORTS
is misconfigured or too small relative to
MAX_*_WORKERS
, jobs can hang or appear serialized.[\[Scaling k8s parallelism\]](https://discuss.airbyte.io/t/826) 4. Replica count of worker pods To increase parallelism, you often also need multiple worker replicas, each able to run up to
MAX_SYNC_WORKERS
jobs.[\[Scaling k8s parallelism\]](https://discuss.airbyte.io/t/826) The scaling guides explicitly recommend increasing
worker.replicaCount
along with the max worker values.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits) 5. Cluster / resource pressure can effectively serialize jobs If your Kubernetes cluster doesn’t have enough CPU/memory to schedule connector pods, syncs can sit pending and run effectively one after another, even though Airbyte’s logical limits are higher.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs) Concrete checks / suggestions: - Verify
SUBMITTER_NUM_THREADS
is set high enough compared to your target concurrency.[\[Scaling k8s parallelism\]](https://discuss.airbyte.io/t/826) - Confirm
worker.maxSyncWorkers
/
worker.maxCheckWorkers
(or
MAX_*_WORKERS
envs) are actually applied to the running worker pods. - Ensure
worker.replicaCount
> 1 if you want higher global concurrency.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits) - Check that
TEMPORAL_WORKER_PORTS
covers at least the total concurrent jobs per worker.[\[Configuring worker\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker) - Look at your node capacity and pod scheduling (many pending
orchestrator/read/write
pods can mean cluster under‑provisioning).[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs) If you can share how you’ve set
SUBMITTER_NUM_THREADS
,
worker.replicaCount
, and your deployment type (Helm values vs env vars), I can help narrow which of these is most likely in your case.
b
I have configured the global logging level to
info
and then changed it to
debug
, but I don't see any changes in the logs. There are no
debug
level log entries showing up. Does anyone know why this might be happening or what I might be missing? @kapa.ai
k
A message from kapa.ai
A few relevant details from the docs and issues: 1. Global log level is controlled by
LOG_LEVEL
Airbyte’s operator docs state that
LOG_LEVEL
defines the log level and defaults to
INFO
, and it must be set to one of the Log4J levels.[\[Configuring Airbyte\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#logging) If you changed a “global logging level” somewhere else (e.g., only in Helm values for a single component or in a config file that isn’t applied), it may not actually be changing the effective
LOG_LEVEL
env var of the running pods. 2. You must restart / redeploy components to pick up the new level The knowledge sources do not explicitly document hot‑reloading of log levels. This usually means that changing
LOG_LEVEL
in values or environment requires restarting the affected pods to take effect. The docs only describe
LOG_LEVEL
as a configuration parameter, not as something you can toggle at runtime.[\[Configuring Airbyte\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#logging) If you edited values (e.g., Helm) but didn’t re‑deploy or roll pods, the logs would stay at the previous level. 3. Beware of
DEBUG
in production
Airbyte explicitly recommends turning off DEBUG logs for non‑testing use, because at scale they can overload the
server
pod and affect the deployment.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#debug-logs) This implies that when
DEBUG
is correctly enabled, you should indeed see significantly more log volume. If you see no difference, it strongly suggests the new level isn’t actually applied. 4. Per‑component vs “global” logging Some references (Helm
values.yaml
snippets, GitHub issues) show log levels being set on specific components like
server.log.level: "DEBUG"
or
workload-launcher.log.level: "INFO"
.[\[Issue 59177](https://github.com/airbytehq/airbyte/issues/59177); \[Issue 60915](https://github.com/airbytehq/airbyte/issues/60915)] If you only changed one component’s level, you’ll see DEBUG logs only from that component, not from others (e.g., connectors, workers). 5. Connector / actor logs are protocol‑based Connector logs are emitted as
AirbyteLogMessage
objects wrapped in
AirbyteMessage
, with a
level
field such as
INFO
,
DEBUG
, etc.[\[AirbyteLogMessage](https://docs.airbyte.com/platform/understanding-airbyte/airbyte-protocol#airbytelogmessage)] If the connector code itself doesn’t emit any DEBUG‑level messages, you won’t see DEBUG lines even if the platform log level is DEBUG. Because the docs don’t provide a step‑by‑step example for switching from INFO to DEBUG and verifying it, there is some uncertainty about your exact setup. But based on the available information, the most likely causes are: - The
LOG_LEVEL
change is not actually applied to the running pods (no restart/redeploy, or changed in the wrong place). - You changed log level only for one component (e.g.,
server
) and are looking at logs for another component that remains at INFO. - The code you’re observing simply doesn’t log at DEBUG level. I’d suggest: 1. Confirm which component’s values you changed (e.g.,
server.log.level
,
worker.log.level
, etc.).[\[Issue 59177](https://github.com/airbytehq/airbyte/issues/59177)] 2. Verify in the pod’s environment that
LOG_LEVEL
is actually set to
DEBUG
for that component.[\[Configuring Airbyte\]](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#logging) 3. Restart or redeploy the affected pods after changing the value. 4. Check a component known to be noisy at DEBUG (e.g.,
server
) and confirm increased log volume, keeping in mind the warning about not leaving DEBUG on in production.[\[Scaling Airbyte\]](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#debug-logs)
b
I’m trying to find the
worker.replicaCount
variable in the
values.yml
file for my Airbyte setup, but I’m not sure if it’s there. Can anyone confirm whether this variable is located in the
values.yml
file, or if it’s configured elsewhere? @kapa.ai
k
A message from kapa.ai
worker.replicaCount
is a value that lives in the Airbyte Helm chart’s
values.yaml
and is used to control how many worker pods are deployed. The scaling docs and example configs show it explicitly under the
worker
section, for both Helm chart v1 and v2, for example:
Copy code
yaml worker:   replicaCount: 2
[[Scaling Airbyte 1.8](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#concurrent-sync-limits); [Scaling Airbyte (latest)](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits)] If you don’t see it in your local
values.yaml
, it may just be omitted and falling back to the chart’s default. In that case you can add a
worker:
section with
replicaCount
yourself, or check the chart’s
values.yaml
on GitHub for the exact structure. [[Helm values ref](https://docs.airbyte.com/platform/deploying-airbyte/values)]
b
We are facing an issue where not all data is being transferred during our sync processes, and it appears to be related to the handling of the
fetch_size
parameter. We have noticed that the number of records transferred in each batch is dynamically adjusted, but this is sometimes leading to incomplete data transfer — some tables are missing data, and not all rows are being transferred. Is there a way to adjust the
fetch_size
manually for better control over how Airbyte handles these large data batches? Can this be configured in the Airbyte UI, or would it require modifying the integration's code? @kapa.ai
k
A message from kapa.ai
Airbyte’s database connectors (MySQL, Postgres, Snowflake, etc.) already use a dynamic
fetchSize
mechanism, and this is not currently user‑configurable from the UI. From the shared discussions: - JDBC/source connectors fetch records in batches whose size is controlled by
fetchSize
. This was changed to be computed dynamically by the connector code.[[MySQL perf](https://discuss.airbyte.io/t/1092); [Sync performance](https://discuss.airbyte.io/t/1578)] - Users explicitly asked whether
fetchSize
can be configured; the answer from the team was “No, it’s not possible”, though the dynamic logic “should now be more clever.”[[Sync jobs speed up](https://discuss.airbyte.io/t/1168)] - For Postgres/Snowflake, support also confirms that
fetchSize
is adjusted dynamically and links performance concerns to an internal GitHub issue, rather than exposing a setting.[[Optimized Postgres](https://discuss.airbyte.io/t/1984); [Sync performance](https://discuss.airbyte.io/t/1578)] There is no mention in the docs or forum excerpts of: - A UI setting to override
fetchSize
. - A documented connector‑level config (YAML or JSON) that lets you set a custom
fetchSize
. So, based on the available information: - You cannot manually tune
fetch_size
via the Airbyte UI or standard connection settings.
- Changing it would require modifying the connector implementation itself (i.e., editing the integration code / building a forked connector), but the exact code locations or patterns are not described in the provided sources. Given the issues you’re seeing (missing rows / incomplete transfer), those forum threads suggest: - Treat it as a bug / performance issue in the specific connector rather than something you can tune away with configuration.[[MySQL missing rows](https://github.com/airbytehq/airbyte/issues/53614); [MySQL perf](https://discuss.airbyte.io/t/1092)] If you share which source connector you’re using (MySQL, Postgres, Snowflake, etc.), the next best step is usually: 1. Capture and share full sync logs with Airbyte support/community so they can check how
fetchSize
is being computed in your runs.[[MySQL perf](https://discuss.airbyte.io/t/1092); [Sync jobs speed up](https://discuss.airbyte.io/t/1168)] 2. Open or link to a GitHub issue for that connector if rows are definitively missing, as that’s treated as a correctness bug, not just a tuning problem.[[MySQL missing rows](https://github.com/airbytehq/airbyte/issues/53614)] The knowledge sources do not show any supported way today to override
fetch_size
manually.
b
"How does Airbyte handle time zones when the source data is in UTC and the server is set to the Europe/Moscow timezone? Will there be any time shifts in columns containing date and time?" @kapa.ai
k
A message from kapa.ai
Airbyte’s behavior depends on how the datetime is typed in the Airbyte record, not on the OS/server timezone: 1. Internal / “technical” columns are always UTC Airbyte’s own columns (like
_airbyte_emitted_at
, timestamps in Avro logical types, etc.) are stored in UTC. For JSON→Avro conversion (used by blob destinations like S3),
date-time
fields become Avro
timestamp-micros
representing “the number of microseconds from the unix epoch, 1 January 1970 000000.000000 UTC”, and the value is stored in UTC regardless of server timezone.[[JSON→Avro rules](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules)] 2. Airbyte temporal types are defined independently of server timezone Airbyte defines five temporal types:
date
,
timestamp_with_timezone
,
timestamp_without_timezone
,
time_with_timezone
,
time_without_timezone
. These are represented as RFC3339 strings with
format: date
or
format: date-time
and disambiguated via
airbyte_type
.[[Data types – specific](https://docs.airbyte.com/platform/understanding-airbyte/supported-data-types#specific-types)] Destinations are expected to honor whether a value is “with timezone” or “without timezone”, and to default to the with‑timezone variant if it’s unspecified.[[Data types – specific](https://docs.airbyte.com/platform/understanding-airbyte/supported-data-types#specific-types)] 3. JSON→Avro / Parquet conversions normalize to UTC For blob destinations, built‑in JSON
date-time
formats are mapped to
timestamp-micros
and “logical type fields will be stored as UTC, respecting timezone as applicable.”[[JSON→Avro rules](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules)] For S3 migration, timestamps with timezone are explicitly converted to UTC; old mixed string+logical unions were removed, but the data semantics remain UTC-based.[[S3 migration – date/time](https://docs.airbyte.com/integrations/destinations/s3-migrations#date-and-time-types)] Implication for your case (source in UTC, server in Europe/Moscow): - If the source sends proper UTC timestamps (e.g.
2024-01-01T12:00:00Z
or
+00:00
), Airbyte treats them as UTC and stores UTC instants. The server’s Europe/Moscow timezone does not cause a shift in those stored values. - Any “shift” you see will usually come from how your destination or BI tool interprets/prints those UTC instants (e.g., automatically displaying them in local time), not from Airbyte changing the underlying moment in time. The provided sources do not describe any behavior where Airbyte re‑bases timestamps based solely on the host/server timezone, so based on current docs, you should not expect an automatic time shift just because the Airbyte server runs in
Europe/Moscow
.
b
"Can Airbyte create a replicated table in ClickHouse? If I create the table myself, will I be able to write data into it, or will Airbyte overwrite it, even if the table name is exactly the same?" @kapa.ai
k
A message from kapa.ai
Airbyte’s ClickHouse destination does not expose any configuration to create
ReplicatedMergeTree
/ replicated tables for you, nor to customize the exact
CREATE TABLE
statement in a way needed for multi‑node replication (for example, adding
ON CLUSTER
or a
ReplicatedMergeTree
engine). A forum post confirms that “today the connector only works with single instance” and that achieving cluster‑wide replicated behavior would require a customized connector implementation.[[Destination to Clickhouse cluster](https://discuss.airbyte.io/t/1026)] Regarding creating the table yourself: - The legacy (v1, archived) ClickHouse destination explicitly creates its own tables with a fixed schema (
_airbyte_ab_id
,
_airbyte_emitted_at
,
_airbyte_data
).[[ClickHouse archived](https://docs.airbyte.com/integrations/destinations/clickhouse-deprecated)] - For the new v2 ClickHouse destination, the docs state that it “writes each stream to its own table” and creates tables in the configured/default database or in the database corresponding to the stream namespace, with typed columns according to the mapping.[[ClickHouse output schema](https://docs.airbyte.com/integrations/destinations/clickhouse#output-schema)] However, the provided sources do not clearly state whether v2 will: - Reuse an existing table if the schema matches, or - Attempt to
CREATE
/
ALTER
and thus fail or modify your pre‑created replicated table. One GitHub issue for the new ClickHouse destination shows that Airbyte does attempt to ALTER tables it created (e.g., changing primary‑key column nullability), and there is currently no configuration to tell it “don’t change destination columns.”[[Clickhouse fails second incremental sync](https://github.com/airbytehq/airbyte/issues/65915)] This strongly suggests that if you pre‑create a replicated table with the same name, Airbyte may still try to manage its schema and could conflict with your engine/cluster settings. So, based on the available information: - Airbyte cannot natively create replicated ClickHouse tables for you. - Pre‑creating a replicated table with the same name is not guaranteed to be safe: the connector may still attempt `CREATE`/`ALTER` operations, and there is no documented option to fully disable that behavior.[[Clickhouse fails second incremental sync](https://github.com/airbytehq/airbyte/issues/65915)] If you need strict control over replicated tables, the current docs and discussions point to building or forking a custom ClickHouse destination that uses your desired
CREATE TABLE
/ engine definitions.[[Destination to Clickhouse cluster](https://discuss.airbyte.io/t/1026)]
b
How does the 'incremental append + deduped' mode work for the ClickHouse connector, and does it increase the load on Airbyte or the destination database? @kapa.ai
k
A message from kapa.ai
For ClickHouse, Incremental | Append + Deduped works the same way as for other destinations, with an implementation optimized for ClickHouse. ### How the mode works In Incremental | Append + Deduped: 1. Incremental Only new or modified records since the last sync are read from the source (first run behaves like a full refresh).[[Sync modes](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes); [Inc. append+dedup overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] 2. Append These incremental records are appended into history tables in the destination; Airbyte keeps the change history there.[[Inc. append+dedup overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] 3. Deduped A final table is produced by: - De‑duplicating on the configured primary key. - Keeping only the latest row per primary key, based on the cursor (e.g.,
updated_at
).[[Inc. append+dedup overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped); [Rules examples](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#rules)] For the ClickHouse destination v2, this mode is explicitly supported, and deduplication leverages
ReplacingMergeTree
on the ClickHouse side.[[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] ### Impact on load Based on the docs: - On Airbyte: - Versus plain Incremental Append, Incremental Append + Deduped adds the work of computing a deduplicated “final” table from the history data.[[Inc. append+dedup overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] - The docs do not quantify the extra CPU/IO, but logically it is more work than Append alone. The knowledge sources don’t provide precise performance numbers. - On ClickHouse: - The connector uses
ReplacingMergeTree
to implement deduplication, so ClickHouse does more merge/dedup work in the background than with pure append tables.[[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] - The docs only advise on choosing an appropriate cursor type for good deduplication (integer, date, or timestamp); otherwise it falls back to
_airbyte_extracted_at
for ordering, which can affect correctness or performance of merges, but they do not provide explicit load benchmarks.[[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] So: - Yes, it inherently adds some extra load compared to simple Incremental Append, on both Airbyte and ClickHouse, due to history storage and dedup/merge work. - The exact magnitude of that load is not specified in the available documentation.
b
Does deduplication happen during the sync process, or is it done afterward by the ClickHouse engine? @kapa.ai
k
A message from kapa.ai
For the new ClickHouse v2 destination, the docs only state that it “leverages ClickHouse’s native
ReplacingMergeTree
for compaction‑style deduplication”
in Incremental Append + Deduped mode, but they do not spell out the exact timing (during the Airbyte sync workflow vs. entirely deferred to ClickHouse merges). [[ClickHouse v2 beta](https://github.com/airbytehq/airbyte/discussions/62035); [ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] From this we can say confidently: - Airbyte configures tables so that deduplication is handled using
ReplacingMergeTree
on the ClickHouse side. [[ClickHouse dest](https://docs.airbyte.com/integrations/destinations/clickhouse)] - The documentation does not clearly describe whether Airbyte also performs additional dedup logic during the sync itself, or relies solely on ClickHouse’s background merge/compaction process. So, based on the available sources, the mechanism involves
ReplacingMergeTree
in ClickHouse, but the exact division of work between “during sync” vs. “afterward by the engine” is not explicitly documented.
b
After the initial synchronization (PostgreSQL source), I see the following error: Failure in source: Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency to prevent this from happening in the future. See @kapa.ai
k
A message from kapa.ai
This error means the Postgres CDC state that Airbyte saved (the “saved offset” / LSN) is older than what Postgres thinks is still safely available in the replication slot (
confirmed_lsn
/
confirmed_flush_lsn
). When that happens, Airbyte can’t continue reading changes incrementally and fails the sync with: > Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency… [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs); [AlloyDB issue](https://github.com/airbytehq/airbyte/issues/48744)] From the docs, this generally occurs because: 1. Required WAL segments were removed - Heavy write activity caused Postgres to purge/archive WAL files needed to continue from Airbyte’s last saved LSN. - This is controlled mainly by
wal_keep_size
(and, on managed services, similar WAL retention settings). [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs); [CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#general-airbyte-configuration)] 2. Syncs are too infrequent for the WAL retention - If your sync interval is longer than the effective WAL retention window, WAL segments will be discarded before Airbyte can read them. [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#general-airbyte-configuration)] 3. Rare case: destination failed after WAL was acknowledged - Source successfully read and acknowledged WAL, but destination failed; on retry, the WAL segment is no longer available because Postgres already considered it consumed. [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs)] In some recent issues, this also shows up after configuration changes or state inconsistencies (e.g., switching tables, removing schemas, or connector behavior around
confirmed_flush_lsn
vs. saved LSN). [[Issue 66091](https://github.com/airbytehq/airbyte/issues/66091); [Issue 55846](https://github.com/airbytehq/airbyte/issues/55846); [Issue 49802](https://github.com/airbytehq/airbyte/issues/49802)] ### What you can do 1. Reset or recreate the connection (required to recover) - With Invalid CDC position behavior set to Re-sync data, Airbyte will trigger a sync from scratch when it detects this mismatch. [[Issue 49802](https://github.com/airbytehq/airbyte/issues/49802)] - Otherwise you may need to Reset connection in the UI, or, in some edge cases (like stale state on removed tables), even delete and recreate the connection, as one user reported. [[Issue 66091](https://github.com/airbytehq/airbyte/issues/66091)] 2. Increase WAL retention / keep WAL longer - Raise
wal_keep_size
(or the managed‑service equivalent like
wal_keep_size
/
max_slot_wal_keep_size
on AlloyDB). [[Postgres troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs); [AlloyDB issue](https://github.com/airbytehq/airbyte/issues/48744)] - Be aware this increases disk usage. 3. **Increase sync frequency so it’s shorter than WAL retention** - Docs recommend that sync frequency must be less than the WAL retention period (e.g., if retention is 3 days, sync at least every 2 days). [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#general-airbyte-configuration)] - For most workloads, daily or every few hours is suggested; for low‑activity DBs, just ensure the interval is well within retention. [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#general-airbyte-configuration)] 4. If you have low write volume and LSN doesn’t advance well - In some cases, WAL disk consumption and LSN behavior with low transaction volumes can be tricky. Docs suggest adding a high‑volume or heartbeat table to the publication and, optionally, using a Debezium heartbeat query to keep the LSN progressing. [[Advanced WAL troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)] 5. If it keeps happening even with high WAL settings and frequent syncs - This may be related to the open bugs around
Saved offset is before replication slot's confirmed lsn
(e.g., mismatch between saved state LSN and
confirmed_flush_lsn
, or behavior on managed services like AlloyDB). [[Issue 48744](https://github.com/airbytehq/airbyte/issues/48744); [Issue 55846](https://github.com/airbytehq/airbyte/issues/55846)] - In that case, collecting logs that show the saved LSN vs
pg_replication_slots.confirmed_flush_lsn
and attaching them to the relevant GitHub issue would be the recommended next step. If you share: - Your Postgres flavor (self‑managed, RDS, AlloyDB, etc.) - Current
wal_keep_size
/ retention settings - Sync frequency - Whether this happens right after the first (snapshot) sync or only after later syncs I can narrow down which of the documented causes is most likely in your setup and which of the above mitigations should be prioritized.
b
We are experiencing an issue with Airbyte. All syncs are turned off, but the server's RAM usage keeps increasing. We don't know how to stop it. What could be the cause? @kapa.ai
k
A message from kapa.ai
This pattern (RAM usage on `airbyte-server`/`airbyte-worker` growing over time even when no syncs are running) matches several known issues and discussions: 1. Known memory / resource leaks in
airbyte-worker
and logging system
- Users have reported that
airbyte-worker
memory keeps increasing over days or weeks, even with a steady or idle workload, sometimes reaching many GB and thousands of threads until the worker must be restarted.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309); [Gradual increase in memory](https://discuss.airbyte.io/t/2249)] - The team acknowledged a suspected memory leak in the logging system, and suggested restarting
airbyte-server
and
airbyte-worker
as a workaround while they investigate.[\[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] - In Kubernetes deployments, users have seen the worker pod’s memory steadily grow from a few hundred MB up to its limit, at which point it gets OOM‑killed; restarting the deployment resets usage temporarily.[\[Memory leak worker pod](https://github.com/airbytehq/airbyte/issues/56944)] 2. Server process consuming large amounts of RAM over time - There are reports of
airbyte-server
using tens of GB of RAM (e.g., 48–50 GB on a 64 GB host) with only a few short‑lived jobs, and memory not being released even when no jobs are running.[\[Unusual RAM usage server](https://discuss.airbyte.io/t/2167)] - Restarting the server temporarily reduces usage, but it grows again when jobs run. The root cause is not fully documented in the provided sources, but it is treated as a bug and tracked on GitHub.[\[Unusual RAM usage server](https://discuss.airbyte.io/t/2167)] 3. Worker memory slowly increasing between syncs - Another user observed that even with a single scheduled sync every few hours, idle memory usage for
airbyte-worker
and other containers kept increasing after each sync and never went back down.[\[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083)] - Support suggested that the worker is doing internal tasks even when no syncs are running, and recommended adding resource constraints as a safeguard (accepting possible Java OOM inside the container instead of exhausting the host), and again pointed to a suspected memory leak.[\[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083); \[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] 4. Thread / logging‑related leaks - One detailed investigation found that the worker had ~30k threads and that most of them came from the log4j S3 appender, with threads blocking on locks and logs not flushing properly to S3; this was characterized as a resource leak in the worker.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309)] - Another forum thread mentions that the team is investigating a memory‑leaky logging system, again aligning with what you’re seeing.[\[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] Given these references, the most likely causes for “RAM keeps increasing even with syncs turned off” are: - A known memory/resource leak in
airbyte-worker
and/or
airbyte-server
, often related to logging and background threads, not strictly tied to active syncs.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309); \[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] - Java processes not releasing heap back to the OS even when they are logically idle; combined with leaks, this looks like ever‑growing RAM usage.[\[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083)] ### What you can do right now Based on the documented workarounds: 1. Restart
airbyte-server
and
airbyte-worker
periodically
- This is the explicit workaround suggested while the memory‑leak issue in the logging system is under investigation.[\[Gradual increase in memory](https://discuss.airbyte.io/t/2249)] - In Kubernetes, some users set memory limits so that hitting the limit causes a pod restart, combined with the container‑orchestrator feature to avoid impacting active syncs.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309)] 2. Add resource limits to isolate the problem - For Docker deployments, set memory limits on
airbyte-worker
(and possibly
airbyte-server
) so they cannot exhaust the host; if limits are too strict you may see Java OOM errors, but you’ll protect the rest of the system.[\[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083)] - For Kubernetes, set reasonable
resources.requests/limits
on the worker pod; users have observed leaks even with limits, but at least the pod will be restarted when it hits the cap.[\[Memory leak worker pod](https://github.com/airbytehq/airbyte/issues/56944)] 3. Monitor which component is growing - Use
docker stats
or
kubectl top pods
/ container metrics to confirm whether it’s specifically
airbyte-server
,
airbyte-worker
, or another pod that’s increasing over time.[\[Gradual increase in memory](https://discuss.airbyte.io/t/2249); \[RAM usage keeps increasing](https://discuss.airbyte.io/t/1083)] - This will help you match your situation to the known issues (e.g., worker vs server vs logging to S3). 4. Check for S3 logging + very high thread counts (if applicable) - If you configured log shipping to S3 and see very high thread counts in the worker, you may be hitting the log4j S3 appender issue.[\[Resource leak in worker](https://discuss.airbyte.io/t/2309)] - In that case, reconsider S3 logging config or contact the team with a thread dump, as that user did. Because the provided sources don’t include a final, merged fix for these leaks, I can’t point to a specific version where it’s definitively resolved. If you haven’t yet: - Check your Airbyte version against the issues above, and - Consider opening or updating a GitHub issue with your version, deployment type, and which container is leaking, referencing those existing reports. That will both help you get guidance on whether a fix exists in a newer version, and give the team more data on how often this still occurs.
b
After a resource leak, we brought down all the containers and are trying to restart them, but we're unable to do so. DEBUG error pulling image Error response from daemon: manifest for airbyte/async-profiler:2.0.1 not found: manifest unknown: manifest unknown DEBUG error pulling image Error response from daemon: manifest for airbyte/utils:2.0.1 not found: manifest unknown: manifest unknown DEBUG found existing image with ref docker.io/kindest/kindnetd:v20250214-acbabc1a DEBUG found existing image with ref docker.io/kindest/local-path-helper:v20241212-8ac705d0 DEBUG found existing image with ref docker.io/kindest/local-path-provisioner:v20250214-acbabc1a DEBUG found existing image with ref registry.k8s.io/coredns/coredns:v1.11.3 DEBUG found existing image with ref registry.k8s.io/etcd:3.5.16-0 DEBUG found existing image with ref registry.k8s.io/kube-apiserver-amd64:v1.32.2 DEBUG found existing image with ref registry.k8s.io/kube-apiserver:v1.32.2 DEBUG found existing image with ref registry.k8s.io/kube-controller-manager-amd64:v1.32.2 DEBUG found existing image with ref registry.k8s.io/kube-controller-manager:v1.32.2 DEBUG found existing image with ref registry.k8s.io/kube-proxy-amd64:v1.32.2 DEBUG found existing image with ref registry.k8s.io/kube-proxy:v1.32.2 DEBUG found existing image with ref registry.k8s.io/kube-scheduler-amd64:v1.32.2 DEBUG found existing image with ref registry.k8s.io/kube-scheduler:v1.32.2 DEBUG found existing image with ref registry.k8s.io/pause:3.10 DEBUG found existing image with ref sha256:85b7a174738baecbc53029b7913cd430a2060e0cbdb5f56c7957d32ff7f241ef DEBUG found existing image with ref sha256:873ed75102791e5b0b8a7fcd41606c92fcec98d56d05ead4ac5131650004c136 DEBUG found existing image with ref sha256:a9e7e6b294baf1695fccb862d956c5d3ad8510e1e4ca1535f35dc09f247abbfc DEBUG found existing image with ref sha256:b6a454c5a800d201daacead6ff195ec6049fe6dc086621b0670bca912efaf389 DEBUG found existing image with ref sha256:baa0d31514ee559c8962d6469a23315bcc8342cbbc1eccc49fa803dcd9653cb7 DEBUG found existing image with ref sha256:bbb6209cc873b9b4095bd014b4687512eea2bd7b246f9ec06f4f6f0be14d9fb6 DEBUG found existing image with ref sha256:c69fa2e9cbf5f42dc48af631e956d3f95724c13f91596bc567591790e5e36db6 DEBUG found existing image with ref sha256:d8e673e7c9983f1f53569a9d2ba786c8abb42e3f744f77dc97a595f3caf9435d DEBUG found existing image with ref sha256:df3849d954c98a7162c7bee7313ece357606e313d98ebd68b7aac5e961b1156f DEBUG found existing image with ref sha256:f1332858868e1c6a905123b21e2e322ab45a5b99a3532e68ff49a87c2266ebc5 DEBUG image does not exist: sha256:61d35bb52ea60d69f9370f423d607b570fed20762c3c48aba104ad4a40bce91f [airbyte/connector-builder-server:2.0.1] DEBUG image does not exist: sha256:de3becfb2fe4b9dacf8f451ba92951572991fac5cbaab9279dbee75276cd336b [airbyte/cron:2.0.1] DEBUG image does not exist: sha256:d8377bdf08f8e0aa5d20daa1fae54f83746f6eefca1cffe49afaf7cc285557ac [airbyte/server:2.0.1] DEBUG image does not exist: sha256:80598943cff75ec039972e968319de0406142fd14b03654ed62dfc93a6e6d97d [airbyte/workload-launcher:2.0.1] DEBUG image does not exist: sha256:1ed225f93e22b417137fd07bf430fc359c9569d4c0b248994ac24790a5ba183d [airbyte/container-orchestrator:2.0.1] DEBUG image does not exist: sha256:3fa68f1b19158e1b8ba6425967d0a99b528f9a0b2d3717bdad64d85c473c5eb7 [airbyte/workload-api-server:2.0.1] DEBUG image does not exist: sha256:56d2130c988ad5dda23c64402f21662545a120c0e6a29ea976a21e5313b6de2a [airbyte/connector-sidecar:2.0.1] DEBUG image does not exist: sha256:865ba61eca5338aaad309a6dad027f142f7dc9c9e4e6b0fde21956c4c061d036 [airbyte/worker:2.0.1] DEBUG image does not exist: sha256:e2a0d56fd627022185cd897093c58d0f330543c803c362560205d558a7619080 [airbyte/bootloader:2.0.1] DEBUG image does not exist: sha256:5df13b07113879aab9ec97e02d8dda972df52b3938d885e1d19b30d310dcf6cb [airbyte/workload-init-container:2.0.1] DEBUG image does not exist: sha256:3e7f68452b6a385f22351573d764fb11fff378352912a0c0eebb5ced9710ce61 [airbyte/db:1.7.0-17] DEBUG image does not exist: sha256:6697cac6ce122978874e3c18368ac1a8b2ad8efb47dfa4d10b2a8523ce636586 [temporalio/auto-setup:1.27.2] DEBUG saving image archive to /tmp/images-tar-4240998277/images.tar DEBUG loading image archive into kind node airbyte-abctl-control-plane ▄ Pulling images (8m23s) @kapa.ai
k
A message from kapa.ai
The errors you’re seeing mean
abctl
is trying to pull images that don’t exist on Docker Hub for 2.0.1, notably: -
airbyte/async-profiler:2.0.1
-
airbyte/utils:2.0.1
Very similar issues have been reported: -
abctl local
/ local setup hanging or failing on
airbyte/async-profiler:1.6.2
or
1.7.2
because those tags don’t exist. [[Local setup issue](https://github.com/airbytehq/airbyte/issues/61339)] - Helm charts 2.0.0 / 2.0.1 were pulled from the repo because they referenced non‑existing images (e.g.,
airbyte/webapp:2.0.0
), and users were instructed to deploy Airbyte 2.0.1 using Helm chart 1.9.1 instead. [[Chart 2.0.0 problems](https://github.com/airbytehq/airbyte/issues/68134); [Use chart 1.9.1](https://github.com/airbytehq/airbyte/issues/68136)] In short: you’ve hit a bad image/tag combination;
abctl
is asking Docker to pull tags that aren’t published. Given the available information, the practical options are: 1. Use a supported chart / version combination If you’re deploying via Helm, follow the guidance to use Helm chart v1.9.1 to deploy Airbyte v2.0.1, not the removed 2.0.x charts that reference missing images.[[Use chart 1.9.1](https://github.com/airbytehq/airbyte/issues/68136); [webapp:2.0.0 not found](https://github.com/airbytehq/airbyte/issues/68134)] 2. Avoid image tags that don’t exist The
async-profiler
tag problem is known: trying to pull
airbyte/async-profiler:1.6.2
or
1.7.2
fails with
manifest unknown
because those tags are not present on Docker Hub.[[Local setup issue](https://github.com/airbytehq/airbyte/issues/61339)] Your logs show the same pattern for
2.0.1
. The knowledge sources do not document a way to override these image names in
abctl local
today, so there is no supported configuration knob to “skip” or change those images in that workflow. 3. If you’re using a local
abctl
install:
- The docs show `abctl`’s
images manifest
being used to list images and then tag/push them to a custom registry, but they assume the images actually exist on Docker Hub for that version. [[Custom image registry](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#configure-a-custom-image-registry)] - When the manifest includes non‑existent tags (as with the
async-profiler
example),
docker pull
simply fails and the install hangs or errors, as in the open GitHub issue.[[Local setup issue](https://github.com/airbytehq/airbyte/issues/61339)] Because the provided sources don’t include a fix for
airbyte/async-profiler:2.0.1
or
airbyte/utils:2.0.1
, they don’t show a way to complete an
abctl local
install that depends on those tags. The documented resolutions in similar cases are: - Switch to a version / chart combination whose images all exist (e.g., Airbyte 1.8 +
--low-resource-mode
resolved the async‑profiler issue for one user). [[Local setup issue](https://github.com/airbytehq/airbyte/issues/61339)] - Update the Helm repo and install with a chart that has correct image references (Helm chart 1.9.1 for Airbyte 2.0.1). [[Use chart 1.9.1](https://github.com/airbytehq/airbyte/issues/68136)] Given the limitations of the current knowledge sources, I can’t point to a specific
abctl
flag or patch that fixes the 2.0.1 `async-profiler`/`utils` tags; the safest documented path is to align your deployment with a version / chart combination that does not reference missing images.
b
What is the correct procedure to follow if I need to add new tables to an existing PostgreSQL publication on the source database? Also, do I need to run the command
ALTER SUBSCRIPTION <subscription_name> REFRESH PUBLICATION;
on the subscriber side, and is it possible to perform this action through Airbyte? @kapa.ai
k
A message from kapa.ai
Airbyte’s docs only cover the publication side for PostgreSQL CDC; they do not describe managing PostgreSQL subscriptions at all. ### 1. Adding new tables to an existing publication For a Postgres CDC source, Airbyte requires that all replicated tables be part of the publication. The documented way to add tables is to run
ALTER PUBLICATION
on the source DB, for example:
Copy code
sql ALTER PUBLICATION <publication_name> ADD TABLE tbl_new1, tbl_new2;
This is the same pattern shown for the initial setup (where you create the publication with specific tables).[[Cloud SQL CDC step 5](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#step-5-create-publication-and-replication-identities-for-each-postgres-table)] So the general procedure on the source: 1. Ensure each new table has an appropriate replication identity:
Copy code
sql    ALTER TABLE tbl_new1 REPLICA IDENTITY DEFAULT;
or
REPLICA IDENTITY FULL
if needed.[[Cloud SQL CDC step 5](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#step-5-create-publication-and-replication-identities-for-each-postgres-table)] 2. Add the new table(s) to the existing publication:
Copy code
sql    ALTER PUBLICATION <publication_name> ADD TABLE tbl_new1, tbl_new2;    ```[[Cloud SQL CDC step 5](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#step-5-create-publication-and-replication-identities-for-each-postgres-table)]  Then, in **Airbyte**:  - Manually **refresh the source schema** on the connection’s *Schema* tab and enable the new stream(s).[[Schema refresh](https://docs.airbyte.com/platform/using-airbyte/schema-change-management#manually-refresh-the-source-schema)] - For CDC, new tables/columns must be snapshotted once before CDC can track changes; use *Refresh Stream* for the new table to trigger that snapshot.[[CDC limitations – adding schemas/columns](https://docs.airbyte.com/platform/understanding-airbyte/cdc#limitations)]  ### 2. `ALTER SUBSCRIPTION ... REFRESH PUBLICATION`  The knowledge sources do **not** mention PostgreSQL subscriptions or the command:  ```sql ALTER SUBSCRIPTION <subscription_name> REFRESH PUBLICATION;
Airbyte’s CDC implementation talks only about publications and how Airbyte reads from them; it does not configure or manage any Postgres subscriptions.[[Cloud SQL CDC step 5](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#step-5-create-publication-and-replication-identities-for-each-postgres-table); [CDC overview](https://docs.airbyte.com/platform/understanding-airbyte/cdc)] Based on the available docs: - Whether you need
ALTER SUBSCRIPTION ... REFRESH PUBLICATION
is not addressed; Airbyte itself does not act as a logical replication subscriber. - There is no documented way in Airbyte to issue that command on your behalf; you’d run any subscription maintenance directly in Postgres, outside of Airbyte. So Airbyte can help by: - Requiring and using a correctly configured publication. - Letting you refresh schema and add new tables/columns inside the Airbyte connection. But it does not manage PostgreSQL subscriptions, and the provided documentation does not describe integrating
ALTER SUBSCRIPTION ... REFRESH PUBLICATION
into Airbyte.
b
Initial load for table public_booking_prices has taken longer than 8 hours, Canceling sync so that CDC replication can catch-up on subsequent attempt, and then initial snapshotting will resume. is is logs from UI airbyte. @kapa.ai
k
A message from kapa.ai
That message is Airbyte’s CDC “Initial Load Timeout” behavior kicking in. With CDC sources (like PostgreSQL), Airbyte separates: - Snapshot phase (initial load) – reads the full table - CDC streaming phase – reads from WAL/logs The Initial Load Timeout (8 hours by default) sets the maximum duration allowed for that snapshot phase. When a table’s initial load exceeds this timeout, Airbyte: 1. Stops the snapshot for that table with a message like the one you see. 2. Captures the current LSN and switches to CDC streaming so it can catch up on WAL changes. 3. On a later attempt, resumes the snapshot from where it left off. [[CDC source config](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] For very large tables (like your
public_booking_prices
), this is expected if the snapshot can’t finish within 8 hours. The docs recommend: - For large / very large databases, increase Initial Load Timeout to 12–24 hours, up to the maximum of 24 hours, based on observed snapshot duration.[[CDC source config](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] So in your case: - The log is not an error in CDC itself, but a timeout safeguard. - If you want the initial snapshot of
public_booking_prices
to complete in a single run, increase Initial Load Timeout in the source’s Advanced CDC settings, then rerun. - If you leave it at 8h, expect the snapshot to progress across multiple attempts while CDC keeps changes up to date.
b
How can I increase the sync timeout to 12 hours? @kapa.ai
k
A message from kapa.ai
To extend that 8‑hour limit to 12 hours you need to increase the Initial Load Timeout setting in your CDC source configuration. From the CDC best‑practices docs: - Initial Load Timeout “sets the maximum duration for the snapshot phase… The maximum allowed value is 24 hours.”[[CDC source config](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] - For large databases they recommend 12–24 hours; default is 8 hours.[[CDC source config](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] For sources that expose this (e.g., Db2 Enterprise explicitly, Postgres via advanced CDC settings): 1. Open your source in Airbyte. 2. Go to the CDC / Advanced section. 3. Find “Initial Load Timeout in Hours” (or similarly named setting). 4. Change it from
8
to
12
and save. [[Db2 CDC config](https://docs.airbyte.com/integrations/enterprise-connectors/source-db2-enterprise#configuring-cdc-in-airbyte); [CDC source config](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] If your specific source connector UI does not expose an Initial Load Timeout parameter, the knowledge sources don’t document another supported way to push the snapshot timeout to 12 hours for that connector.
b
I don't understand where this setting is located, Initial Waiting Time. Maybe it's in CheLM? @kapa.ai
k
A message from kapa.ai
Initial Waiting Time
is not a global Helm/CheLM setting; it’s a per‑connection, per‑source option exposed in the connector configuration UI for CDC‑capable sources (for example, Postgres, Db2 Enterprise, MongoDB v2, SAP HANA, etc.). From the docs: - For Postgres, “Initial Waiting Time in Seconds (Advanced)” is an advanced CDC parameter that controls how long the connector waits for WAL / CDC events before timing out.[[Postgres CDC troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-setting-up-initial-cdc-waiting-time); [Postgres issue](https://github.com/airbytehq/airbyte/issues/50435)] - CDC best‑practices describe Initial Waiting Time as a source configuration setting that affects both the snapshot and CDC incremental phases; the range “varies by source (check your source configuration page).”[[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices); [Next CDC best practices](https://docs.airbyte.com/platform/next/understanding-airbyte/cdc-best-practices)] You can see similar fields in other CDC‑style sources: - MongoDB v2 lists
initial_waiting_seconds
as “Initial Waiting Time in Seconds (Advanced)” in its config fields.[[MongoDB v2 reference](https://docs.airbyte.com/integrations/sources/mongodb-v2#reference)] - Db2 Enterprise and SAP HANA CDC expose related advanced timing options (e.g., Initial Load Timeout, Initial Load Timeout in Hours, etc.) in the connector configuration, not in Helm values.[[Db2 CDC config](https://docs.airbyte.com/integrations/enterprise-connectors/source-db2-enterprise#configuring-cdc-in-airbyte); [SAP HANA CDC](https://docs.airbyte.com/integrations/enterprise-connectors/source-sap-hana#change-data-capture-cdc)] So, to change Initial Waiting Time: 1. Open your source in the Airbyte UI. 2. Click Edit configuration. 3. Scroll to the Advanced / CDC section. 4. Look for “Initial Waiting Time in Seconds (Advanced)” (or similarly named) and adjust it within the allowed range (for Postgres: 120–1200 seconds).[[Postgres CDC troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-setting-up-initial-cdc-waiting-time)] The provided knowledge sources do not show any
values.yaml
(Helm) key for
Initial Waiting Time
, so there’s no documented way to set it globally via Helm; it’s configured per connector in the UI.
b
How long does Airbyte keep logs by default and what is the limit? And can I adjust these values? @kapa.ai
k
A message from kapa.ai
By default, Airbyte does keep and clean up logs automatically, but the exact defaults differ by version and the mechanism has changed over time. ### 1. Current/log-based retention control In recent versions, log and workspace cleanup is tied to Temporal history retention: - The environment variable
TEMPORAL_HISTORY_RETENTION_IN_DAYS
“defines the retention period of the job history in Temporal” and defaults to 30 days.[\[Configuring Airbyte 1.8](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-airbyte#data-retention); [Configuring Airbyte (next)](https://docs.airbyte.com/platform/next/operator-guides/configuring-airbyte#data-retention)] - Forum answers explicitly recommend changing this variable to reduce storage usage and help clean Temporal history/log-related data (for both Docker and K8s deployments).[\[Forum 2200](https://discuss.airbyte.io/t/2200); [Forum 2417](https://discuss.airbyte.io/t/2417); [GitHub discussion 55233](https://github.com/airbytehq/airbyte/discussions/55233); [Issue 55812](https://github.com/airbytehq/airbyte/issues/55812)] So today: - Default retention window: 30 days of Temporal job history. - You can adjust it by setting
TEMPORAL_HISTORY_RETENTION_IN_DAYS
to the number of days you want. ### 2. Older / legacy workspace log sweeping Older scheduler‑based deployments used: -
MINIMUM_WORKSPACE_RETENTION_DAYS
(default 1 day) -
MAXIMUM_WORKSPACE_RETENTION_DAYS
(default 60 days) -
MAXIMUM_WORKSPACE_SIZE_MB
(default 5 GB) to control deletion of sync logs in the workspace.[\[Rotation logs](https://discuss.airbyte.io/t/332); [Log sweeping](https://discuss.airbyte.io/t/2714)] However: - That mechanism was removed with the old Scheduler, and the documented replacement is again
TEMPORAL_HISTORY_RETENTION_IN_DAYS
.[\[Log sweeping](https://discuss.airbyte.io/t/2714); [Forum 2200](https://discuss.airbyte.io/t/2200); [Forum 2417](https://discuss.airbyte.io/t/2417)] ### 3. Practical answer to your questions - How long are logs kept by default? Airbyte keeps job history (and associated Temporal log history) for 30 days by default via
TEMPORAL_HISTORY_RETENTION_IN_DAYS
.[\[Configuring Airbyte 1.8](https://docs.airbyte.com/platform/1.8/operator-guides/configuring-airbyte#data-retention)] - Is there a size limit? Older scheduler‑based setups used a 5 GB workspace size limit (
MAXIMUM_WORKSPACE_SIZE_MB
default 5 GB) as part of log sweeping,[\[Rotation logs](https://discuss.airbyte.io/t/332)] but this config was removed and the current docs do not describe an equivalent size‑based limit in the new system. The knowledge sources do not clearly state a current global size cap beyond that legacy behavior. - Can you adjust these values? - Yes: you can adjust retention in days by setting
TEMPORAL_HISTORY_RETENTION_IN_DAYS
(for example, to 7 or 1 day).[\[Configuring Airbyte](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#data-retention); [Issue 55812](https://github.com/airbytehq/airbyte/issues/55812)] - The older
MINIMUM_WORKSPACE_RETENTION_DAYS
,
MAXIMUM_WORKSPACE_RETENTION_DAYS
, and
MAXIMUM_WORKSPACE_SIZE_MB
are no longer effective in current deployments using Temporal.[\[Log sweeping](https://discuss.airbyte.io/t/2714)] If you need stricter disk control (e.g., object storage for logs filling up), the sources also show operators manually deleting old log objects (e.g., in MinIO) in addition to tuning
TEMPORAL_HISTORY_RETENTION_IN_DAYS
.[\[Issue 55812](https://github.com/airbytehq/airbyte/issues/55812)]
b
What could be the problem? I can run sync in Airbyte, but I can't create new connections, getting the error "NetworkError when attempting to fetch resource." At the same time, I'm the only one in the team facing this issue. @kapa.ai
k
A message from kapa.ai
Most likely this is a client‑side / browser‑side problem rather than an Airbyte server issue, since: - You can still run syncs, which means the backend is healthy. - Other team members are not affected, so the server and network are working for them. Very similar “can’t create new connection / UI breaks, but backend OK” cases in the docs and issues were caused by: 1. Browser cache / stale frontend assets In Airbyte OSS 1.6.0, users saw “Unable to save the connection” with no visible error and no server issue; it turned out to be a browser caching problem of form data / JS assets. Clearing the browser’s application data (cache, storage) fully fixed it.[[New connection broken](https://github.com/airbytehq/airbyte/issues/58110)] Your
NetworkError when attempting to fetch resource
message fits the pattern of the browser trying to call an old or invalid endpoint. 2. Frontend calling a wrong / non‑existent endpoint For example, in 1.7.1 the Connector Builder UI called
/api/v1/connector_builder/manifest/resolve
, which didn’t exist in that version and always returned 404, breaking only that feature.[[Connector Builder 404](https://github.com/airbytehq/airbyte/issues/63770)] In that case, the backend was fine; the bug was isolated to one UI path. 3. Local networking / DNS / proxy issues specific to one machine Other forum threads about “Cannot reach server” or failing connection tests point to DNS / firewall / proxy issues when only some calls from the browser fail while the Airbyte instance itself is healthy.[[Google Sheets DNS](https://discuss.airbyte.io/t/2626); [GKE cannot reach server](https://discuss.airbyte.io/t/763)] Given the available information: - The Airbyte backend is likely OK (syncs run). - The issue is likely your browser or local network environment. What to try (in order): 1. Hard‑refresh / clear cache and storage for the Airbyte URL - Fully clear site data (cache, local storage, indexedDB) for your Airbyte URL and reload, as done in the 1.6.0 issue.[[New connection broken](https://github.com/airbytehq/airbyte/issues/58110)] 2. Try a different browser or incognito window on your machine - If that works, it confirms a cache/extension issue in your main browser. 3. Check the browser DevTools → Network tab - Look at the failing request when you open the “new connection” page: status code, URL. This is what support asked for in similar “cannot reach server / failed schema fetch” issues.[[GitHub schema fetch](https://discuss.airbyte.io/t/905); [GKE cannot reach server](https://discuss.airbyte.io/t/763)] 4. Compare with a teammate’s machine - If they can create connections, have them open DevTools and compare the working request URL and response to your failing one. The knowledge sources don’t show a server‑side Airbyte bug that matches “only one user gets NetworkError on new connection while syncs still work,” so the evidence points strongly to local browser cache or network configuration on your machine.
b
Can Airbyte be converted to local time? @kapa.ai
k
A message from kapa.ai
Airbyte itself does not provide a global “convert everything to local time” switch. Its documented behavior is: - Airbyte defines explicit temporal types:
date
,
timestamp_with_timezone
,
timestamp_without_timezone
,
time_with_timezone
,
time_without_timezone
, all represented as RFC 3339 strings plus an
airbyte_type
annotation to specify whether the value includes a timezone.[[Data types](https://docs.airbyte.com/platform/understanding-airbyte/supported-data-types#specific-types)] - For blob destinations (Avro/Parquet), JSON
date-time
values are converted to Avro
timestamp-micros
and are stored as UTC, “respecting timezone as applicable.”[[JSON–Avro](https://docs.airbyte.com/platform/understanding-airbyte/json-avro-conversion#conversion-rules)] - Some destinations (for example, GCS Data Lake / Iceberg) explicitly convert “time with timezone” and “timestamp with timezone” to UTC before writing, by design.[[GCS Data Lake schema](https://docs.airbyte.com/integrations/destinations/gcs-data-lake#output-schema)] So: - Airbyte generally normalizes time‑with‑timezone data to UTC at the storage level. - There is no documented setting to automatically convert all timestamps into an arbitrary local timezone (e.g., Europe/Moscow) during sync. - If you need local time, you typically handle the conversion in the destination system or downstream queries/BI tools, not in Airbyte itself. The knowledge sources do not describe any feature that globally “converts Airbyte to local time,” so any such behavior would need to be implemented outside of Airbyte or via custom connector logic.
b
"How can we achieve parallelism in Airbyte, which configurations need to be changed, given that it was deployed via abctl? We have 3 worker replicas, but still only one sync is processed at a time. The
max_sync_workers
configuration is currently set to the default." @kapa.ai
k
A message from kapa.ai
To get real parallelism you need to adjust per‑worker limits and sometimes global limits, not just the worker replica count. From the docs and forum threads, concurrency is controlled by: 1. Per‑worker sync limit Each worker pod can only run up to
MAX_SYNC_WORKERS
sync jobs at once. With
abctl
(Kubernetes under the hood), this is set via env or Helm values:
Copy code
yaml worker:   maxSyncWorkers: 10   # example   maxCheckWorkers: 5
or, in env form:
Copy code
yaml MAX_SYNC_WORKERS=10 MAX_CHECK_WORKERS=5
Defaults are 10 sync workers and 5 check workers per worker container.[[Config worker](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker)] With 3 worker replicas and default
MAX_SYNC_WORKERS=10
, you can run up to 30 syncs in parallel in theory, but other limits may be capping you. 2. Global concurrency / scheduler limits Older docs and forum answers mention: -
SUBMITTER_NUM_THREADS
– hard upper limit on total concurrent jobs across the system (deprecated with the new Temporal scheduler, but still referenced in some setups).[[Scaling k8s parallelism](https://discuss.airbyte.io/t/826); [How to configure concurrency](https://discuss.airbyte.io/t/1077)] - With the new scheduler, concurrency is effectively:
parallel syncs ≈ number of worker pods × MAX_SYNC_WORKERS
(also bounded by available ports).[[How to configure concurrency](https://discuss.airbyte.io/t/1077)] If
SUBMITTER_NUM_THREADS
is still low in your environment, it can serialize syncs even if you have multiple workers and higher
MAX_SYNC_WORKERS
.[[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)] 3. Temporal worker ports Each worker pod needs enough ports to talk to job pods: -
TEMPORAL_WORKER_PORTS
defines the local ports the worker uses; Helm exposes 9001–9040 by default (40 ports).[[Config worker](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker)] - In practice, this caps how many jobs a single worker can run; forum guidance suggests something like:
parallel syncs per worker ≈ min(MAX_SYNC_WORKERS, TEMPORAL_WORKER_PORTS/4)
.[[How to configure concurrency](https://discuss.airbyte.io/t/1077); [Scaling k8s parallelism](https://discuss.airbyte.io/t/826)] If
TEMPORAL_WORKER_PORTS
is too small, you won’t get the parallelism implied by
MAX_SYNC_WORKERS
. 4. Cluster resources If your Kubernetes cluster is under‑provisioned (CPU/memory), pods for additional syncs may stay pending, so you effectively see only one running at a time even though Airbyte wants to run more.[[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)] --- ### What to change / check in your case Given you deployed with
abctl
and have 3 worker replicas but only 1 sync at a time: 1. Increase
MAX_SYNC_WORKERS
on workers
Set it explicitly higher than 1 (e.g., 5 or 10) via the worker config (Helm values or env), then restart/redeploy workers so the new value is applied.[[Config worker](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker)] 2. Ensure global limits aren’t 1 If your environment still uses
SUBMITTER_NUM_THREADS
, make sure it’s not set to 1; forum examples for ~100 concurrent syncs use values like:
Copy code
text SUBMITTER_NUM_THREADS = 200 MAX_SYNC_WORKERS      = 100 Number of workers     = 3
[[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)] 3. Check
TEMPORAL_WORKER_PORTS
Confirm it exposes enough ports (e.g., 9001–9040 or more). If you’ve customized it to a very small range, that can cap concurrency per worker.[[Config worker](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#worker); [How to configure concurrency](https://discuss.airbyte.io/t/1077)] 4. Verify cluster capacity Use
kubectl get pods
/ metrics to see if additional sync pods (
orchestrator
,
read
,
write
) are pending due to lack of resources. If so, you’ll need more node capacity or lower per‑job resource requests.[[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)] Because
abctl
specifics (exact flags/fields) are not detailed in the provided sources, I can’t give you the exact file/command to edit, but the knobs you need to adjust are: -
worker.replicaCount
(you already have 3) -
MAX_SYNC_WORKERS
/
worker.maxSyncWorkers
- Possibly
SUBMITTER_NUM_THREADS
(if present) -
TEMPORAL_WORKER_PORTS
- Underlying K8s resources If you share how you’re configuring Airbyte with
abctl
(values file vs defaults), you can map these directly into that config.
b
Here is what we changed, the rest of the parameters are set to default. worker: maxSyncWorkers: 10 replicaCount: 3 resources: requests: cpu: 100m memory: 100Mi limits: cpu: 3 memory: 4Gi workloadLauncher: resources: requests: cpu: 100m memory: 100Mi limits: cpu: 3 memory: 4Gi temporal: resources: requests: cpu: 100m memory: 100Mi limits: cpu: 3 memory: 4Gi server: resources: requests: cpu: 100m memory: 100Mi limits: cpu: 3 memory: 4Gi workloadApiServer: resources: requests: cpu: 100m memory: 100Mi limits: cpu: 3 memory: 4Gi @kapa.ai
k
A message from kapa.ai
Based on the knowledge sources, there are a few important points that explain why you might still see only one sync at a time, even with the config you shared: 1.
maxSyncWorkers
is only one of the limits
You correctly set:
Copy code
yaml worker:   maxSyncWorkers: 10   replicaCount: 3
So in theory you can run up to 30 syncs in parallel. But concurrency is also constrained by: - The number of worker pods (
replicaCount
) - The per‑worker max (
maxSyncWorkers
) - Temporal worker ports (
TEMPORAL_WORKER_PORTS
) - Global scheduler behavior and available cluster resources [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)]. 2. You haven’t changed
TEMPORAL_WORKER_PORTS
In Kubernetes deployments, each worker can only run as many jobs as it has Temporal worker ports. The docs and forum guidance describe that concurrency per worker is effectively: > parallel syncs per worker ≈ min(
MAX_SYNC_WORKERS
,
TEMPORAL_WORKER_PORTS
/ 4) [[Scaling k8s parallelism](https://discuss.airbyte.io/t/826); [How to configure concurrency](https://discuss.airbyte.io/t/1077)]. If
TEMPORAL_WORKER_PORTS
is left at a very small default or misconfigured, it can cap you to 1 sync per worker even though
maxSyncWorkers
is 10. The docs don’t show the exact default in your abctl setup, so this is an area of uncertainty, but it is explicitly called out as affecting concurrency [[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)]. 3. Cluster resources can still serialize jobs Even with your resource limits, if the Kubernetes cluster doesn’t have enough free CPU/memory to schedule the
orchestrator
,
read
, and
write
pods for multiple syncs, you’ll effectively see only one running at a time. The scaling docs note that each concurrent sync needs at least 3 extra pods, and under‑provisioned clusters often manifest as “stuck” or effectively serialized jobs [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)]. 4. Other concurrency knobs are at defaults The knowledge sources mention older/global limits like
SUBMITTER_NUM_THREADS
in some Kubernetes setups, which can cap total concurrent jobs regardless of
MAX_SYNC_WORKERS
[[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)]. However, they don’t show how (or if) this is exposed in your abctl‑based deployment, so I can’t say for sure whether it’s affecting you. --- ### What you can try next (based on the docs) 1. Check / increase
TEMPORAL_WORKER_PORTS
Ensure it covers enough ports per worker so that
TEMPORAL_WORKER_PORTS / 4
is at least 10, to match your
maxSyncWorkers
target [[Scaling k8s parallelism](https://discuss.airbyte.io/t/826)]. 2. Verify that multiple sync pods are actually being scheduled Use
kubectl get pods
and look for multiple
orchestrator-
,
read-
, and
write-
pods in
Running
state at the same time. If they’re
Pending
, you likely need more cluster capacity or lower per‑job resource defaults [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-syncs)]. 3. Confirm that your
maxSyncWorkers
is applied
Make sure the running worker pods actually have
MAX_SYNC_WORKERS=10
in their environment (e.g., via
kubectl describe pod
). The scaling docs assume this env is what drives concurrency [[Scaling Airbyte](https://docs.airbyte.com/platform/enterprise-setup/scaling-airbyte#concurrent-sync-limits)]. The knowledge sources don’t provide abctl‑specific YAML examples beyond the generic Helm values, so I can’t give an exact abctl command, but the main missing piece from what you shared is likely the Temporal worker ports and/or actual cluster capacity, not the
maxSyncWorkers
setting itself.