https://linen.dev logo
Join Slack
Powered by
airbyuyte didn't load everything when I refreshed ...
# replication-ask-ai
r
airbyuyte didn't load everything when I refreshed a big table to load whole table again
k
There are a few known issues and considerations that could explain why Airbyte didn't load all rows when refreshing a large table: Possible Causes 1. CTID Scanning Race Condition (PostgreSQL sources) If your source is PostgreSQL, there is a known bug where full refresh syncs using CTID-based batch scanning can silently drop rows on tables with concurrent writes. [source-postgres issue] Here's what happens: • Airbyte scans the table in batches using physical page ranges (CTIDs) • If rows are updated during the scan, PostgreSQL's MVCC creates new physical row versions at different locations • A row's old version may be "dead" in a later batch's snapshot, while its new version landed in a range already scanned by an earlier batch • The row is silently dropped with no error or warning in the logs This has been observed in production with ~1.1M row tables scanned over multiple batches. 2. Large Table Sync Failures (MySQL sources) If your source is MySQL, there is a reported issue where syncing large tables (e.g., 38+ GB) results in missing rows, even when using CDC/binary logs. [source-mysql issue] 3. At-Least-Once Delivery, Not Exactly-Once Airbyte is an at-least-once delivery system. While this means records can be duplicated, it also means that in certain failure scenarios, a retry may be needed to capture all records. [Resumability docs] Recommendations • For PostgreSQL with concurrent writes: Switch to Incremental | Append + Dedupe with a cursor column (e.g., 
updated_at
) to avoid CTID scanning issues. [source-postgres issue] • For very large tables in general: Full refresh syncs copy the entire dataset every time, which can be infeasible for large tables. Incremental sync modes are generally more appropriate. [Full Refresh docs] • Check your logs for any error messages indicating rows that failed to sync. Could you share more details about your source connector (PostgreSQL, MySQL, etc.) and the Airbyte version you're using? That would help narrow down the specific cause. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai it is mysql. can I increase number of rows chunks per file? will it improve things?
k
Based on the knowledge sources available, there is some discussion about 
fetchSize
(the batch size for reading rows from MySQL) in the context of MySQL sync performance. The 
fetchSize
parameter controls how many rows are fetched per batch from the source database. Airbyte introduced a dynamic 
fetchSize
computation in the MySQL connector to try to maximize throughput while preventing memory issues. [MySQL performance forum] However, testing showed that tweaking 
fetchSize
did not produce significant performance improvements in practice — one user found virtually no difference between connector versions with and without dynamic 
fetchSize
, with syncs taking ~8h 22m vs ~8h 23m for the same table. [MySQL performance forum] Regarding your specific issue of missing rows on large MySQL tables, the knowledge sources do not mention that increasing chunk/batch size resolves the missing data problem. The missing rows issue on large MySQL tables (38+ GB) has been reported as an open bug, and the suggested debugging steps were: [source-mysql issue] • Check if the missing rows contain unusually large column objects or long strings • Review sync logs for any mention of skipped rows • Consider using Docker or Kubernetes (kubectl) installation instead of 
abctl
, as 
abctl
may have limitations with large-scale data transfers One community member also noted that splitting the sync into multiple connections (one per stream/table) can help parallelize work, though this doesn't directly address missing rows. [sync performance forum] The knowledge sources don't provide a direct configuration option to increase rows-per-chunk that is confirmed to fix the missing data issue for MySQL. I'd recommend checking your sync logs carefully for skipped row errors and ensuring you're on a recent connector version. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai maybe my problem of not load all rows is regarding a limit of resource?
Copy code
2026-04-15 10:33:07 info [initContainer] image: airbyte/workload-init-container:1.8.5 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
2026-04-15 10:33:07 info Launching replication pod: replication-job-169541-attempt-0 (selectors = {}) with containers:
2026-04-15 10:33:07 info [source] image: airbyte/source-mysql:3.1.1 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
2026-04-15 10:33:07 info [destination] image: airbyte/destination-snowflake:3.15.4 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
2026-04-15 10:33:07 info [orchestrator] image: airbyte/container-orchestrator:1.8.5 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
k
Resource limits are definitely worth investigating as a potential cause. Looking at your logs, your source container has: • Memory limit: 4Gi • Memory request: 1Gi According to Airbyte's documentation, the Source worker is the main memory culprit in a sync job, because it reads up to 10,000 records in memory at a time. For tables with large row sizes, this can be a significant constraint — for example, a table with an average row size of 0.5MB would require 5GB of RAM just for that in-memory buffer. [Scaling Airbyte docs] With a 38+ GB MySQL table, a 4Gi memory limit on the source container could plausibly cause issues if your rows are large. What You Can Do You can increase the resource limits for your connector pods. Airbyte supports configuring connector resource requirements. The documentation points to the Configuring connector resources guide for details on how to do this. [Scaling Airbyte docs] For example, one user dealing with large CDC syncs set connection-level resource requirements like this (via direct DB update on Kubernetes):
Copy code
UPDATE connection 
SET resource_requirements = '{"cpu_limit": "1.5", "cpu_request": "1.0", "memory_limit": "20Gi", "memory_request": "13Gi"}' 
WHERE id = '<your-connection-id>';
[Java heapspace issue] However, I should note that even with increased memory, the missing rows issue on large MySQL tables has been reported as an open bug that isn't solely explained by resource limits. [source-mysql issue] So increasing resources may help but might not fully resolve the problem. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai if I set no limit of resources? also after the database update, do I need to do abctl local install ?
k
Setting No Resource Limits If you set no resource limits (i.e., leave 
JOB_MAIN_CONTAINER_MEMORY_LIMIT
and related variables unset), the defaults are none — meaning no explicit limit is applied to the containers. [Configuring Connector Resources] This could allow the containers to use more memory if the node has it available, which may help with large table syncs. After the Database Update — Do You Need to Re-run abctl local install? The knowledge sources do not explicitly state whether a re-run of 
abctl local install
is required after updating 
resource_requirements
directly in the database. However, based on what's documented: • The 
connection.resource_requirements
database update takes effect at the connection level and is read by the workload-launcher when it spawns new replication pods. [Configuring Connector Resources] • Users who applied the SQL update reported it worked without any reinstallation — they simply triggered a new sync. [workload resource issue] So you likely do not need to re-run 
abctl local install
 after a database-level 
resource_requirements
update — just trigger a new sync and the new pod should pick up the updated values. > ⚠️ Important caveat: There is an open bug report indicating that resource configurations (including 
values.yaml
, env vars, and connection overrides) may be ignored by 
abctl
v2.0.1. [abctl resource limits issue] This is under investigation by the Airbyte engineering team, so results may vary. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai SET resource_requirements = '{"cpu_limit": "1.5", "cpu_request": "1.0", "memory_limit": "20Gi", "memory_request": "13Gi"}' bash: UPDATE: command not found bash: SET: command not found
k
It looks like you're trying to run the SQL command directly in your terminal/bash shell instead of inside a PostgreSQL client. The 
UPDATE ... SET ...
is a SQL statement, not a bash command. You need to connect to the Airbyte PostgreSQL database first, then run the SQL. Here's how to do it with `abctl`: Step 1: Connect to the Airbyte database
Copy code
abctl local credentials
This will give you the database credentials. Then connect using `psql`:
Copy code
psql -U airbyte -d db-airbyte
Or in one step:
Copy code
psql -U airbyte -d db-airbyte -c "UPDATE connection SET resource_requirements = '{\"cpu_limit\": \"1.5\", \"cpu_request\": \"1.0\", \"memory_limit\": \"20Gi\", \"memory_request\": \"13Gi\"}' WHERE id = '<your-connection-id>';"
Step 2: Find your connection ID Your connection ID is in the Airbyte UI URL. The format is: 
<base_url>/workspaces/<workspace-id>/connections/<connection-id>/status
[Configuring Connection-Specific Requirements] Once connected to 
psql
, run the SQL statement as-is — it will work inside the PostgreSQL prompt, not in bash. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai PRD [glpd@dnlam2vp-glmis-tac32 .airbyte]$ docker exec -it airbyte-abctl-control-plane /bin/bash root@airbyte-abctl-control-plane:/# kubectl exec -it -n airbyte-abctl airbyte-db-0 -- /bin/bash psql -U airbyte db-airbyte airbyte-db-0:/$ UPDATE connection SET resource_requirements = '{"cpu_limit": "1.5", "cpu_request": "1.0", "memory_limit": "20Gi", "memory_request": "13Gi"}' bash: UPDATE: command not found bash: SET: command not found
k
You're very close! The issue is that you ran 
psql
but it seems the shell didn't enter the PostgreSQL interactive prompt — the 
UPDATE
command was interpreted by bash instead of psql. Notice in your session you ran:
Copy code
psql -U airbyte db-airbyte
airbyte-db-0:/$ UPDATE connection ...
The 
airbyte-db-0:/$
prompt means you're still in bash, not inside psql. The 
psql
command may have exited immediately or failed silently. Try this instead — use the 
-c
flag to pass the SQL directly:
Copy code
psql -U airbyte db-airbyte -c "UPDATE connection SET resource_requirements = '{\"cpu_limit\": \"1.5\", \"cpu_request\": \"1.0\", \"memory_limit\": \"20Gi\", \"memory_request\": \"13Gi\"}' WHERE id = '<your-connection-id>';"
Or, to enter the interactive psql prompt properly:
Copy code
psql -U airbyte db-airbyte
Wait for the 
airbyte=#
prompt (not 
airbyte-db-0:/$
), then paste:
Copy code
UPDATE connection
SET resource_requirements = '{"cpu_limit": "1.5", "cpu_request": "1.0", "memory_limit": "20Gi", "memory_request": "13Gi"}'
WHERE id = '<your-connection-id>';
Your connection ID can be found in the Airbyte UI URL: 
<base_url>/workspaces/<workspace-id>/connections/<connection-id>/status
[Configuring Connection-Specific Requirements] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai should I change it as well? PRD [glpd@dnlam2vp-glmis-tac32 .airbyte]$ docker exec -it airbyte-abctl-control-plane /bin/bash root@airbyte-abctl-control-plane:/# kubectl exec -n airbyte-abctl deploy/airbyte-abctl-workload-launcher -- env | grep -E "JOB_MAIN|SIDECAR" JOB_MAIN_CONTAINER_CPU_REQUEST=0.5 CONNECTOR_SIDECAR_IMAGE=airbyte/connector-sidecar:1.8.5 JOB_MAIN_CONTAINER_MEMORY_REQUEST=512Mi JOB_MAIN_CONTAINER_MEMORY_LIMIT=1Gi JOB_MAIN_CONTAINER_CPU_LIMIT=1 root@airbyte-abctl-control-plane:/#
k
Based on the conversation history, you're asking whether you should also update the 
actor_definition
table (connector definition level) in addition to the 
connection
table. The answer depends on your goal: • 
connection
table update — applies only to this specific connection (highest precedence, overrides everything else). This is what you've been doing. [Configuring Connection-Specific Requirements] • 
actor_definition
table update — applies to all connectors of that type (e.g., all MySQL sources). You'd use this if you want the resource increase to apply to every MySQL sync, not just one connection. [Configuring Connector Resources] For your case (one large MySQL table causing issues), updating just the 
connection
table is the right approach — it's more targeted and won't affect other connections unintentionally. If you later find that all your MySQL sources need more resources, you could also update 
actor_definition
with:
Copy code
UPDATE actor_definition 
SET resource_requirements = '{"jobSpecific": [{"jobType": "sync", "resourceRequirements": {"cpu_limit": "1.5", "cpu_request": "1.0", "memory_limit": "20Gi", "memory_request": "13Gi"}}]}'
WHERE id = '<actor-definition-id>';
But for now, the connection-level update you're working on should be sufficient. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
Yes, you should update these environment variables. Your workload-launcher currently has very low instance-wide defaults: • 
JOB_MAIN_CONTAINER_MEMORY_LIMIT=1Gi
• 
JOB_MAIN_CONTAINER_MEMORY_REQUEST=512Mi
• 
JOB_MAIN_CONTAINER_CPU_LIMIT=1
• 
JOB_MAIN_CONTAINER_CPU_REQUEST=0.5
These are the instance-wide resource settings that apply to all job containers as a baseline. [Configuring Connector Resources] However, since you've already set 
resource_requirements
directly on the 
connection
table (connection-specific level), that takes highest precedence and overrides these instance-wide values. So your large MySQL sync should already be using the 20Gi/13Gi memory you set at the connection level, not the 1Gi limit from the env vars. That said, it's still worth updating the instance-wide values to something more reasonable as a baseline, so other connections aren't unnecessarily constrained. You can do this by adding 
extraEnv
to your 
values.yaml
for the workload-launcher:
Copy code
workloadLauncher:
  extraEnv:
    - name: JOB_MAIN_CONTAINER_CPU_LIMIT
      value: "3"
    - name: JOB_MAIN_CONTAINER_CPU_REQUEST
      value: "1"
    - name: JOB_MAIN_CONTAINER_MEMORY_LIMIT
      value: "4Gi"
    - name: JOB_MAIN_CONTAINER_MEMORY_REQUEST
      value: "2Gi"
[abctl resource limits issue] Bottom line: For your specific large MySQL connection, the database-level 
resource_requirements
update you already made is what matters most. The env vars are the fallback for all other connections. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai using the YAML don't work, how I change it directly ?
k
Based on the conversation history, you want to change the 
JOB_MAIN_CONTAINER_*
environment variables directly on the workload-launcher deployment, bypassing 
values.yaml
. You can patch the deployment directly with `kubectl`:
Copy code
kubectl set env -n airbyte-abctl deploy/airbyte-abctl-workload-launcher \
  JOB_MAIN_CONTAINER_CPU_REQUEST=1 \
  JOB_MAIN_CONTAINER_CPU_LIMIT=3 \
  JOB_MAIN_CONTAINER_MEMORY_REQUEST=2Gi \
  JOB_MAIN_CONTAINER_MEMORY_LIMIT=4Gi
This will trigger a rollout of the workload-launcher pod with the new env vars. You can verify it worked with the same command you used before:
Copy code
kubectl exec -n airbyte-abctl deploy/airbyte-abctl-workload-launcher -- env | grep -E "JOB_MAIN|SIDECAR"
> ⚠️ Important reminder: As noted in the previous conversation, there is an open bug where these instance-wide env vars may still be ignored by 
abctl
deployments. [abctl resource limits issue] The connection-level 
resource_requirements
database update you already made takes highest precedence and is the most reliable workaround confirmed to work by other users. [Configuring Connector Resources] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai root@airbyte-abctl-control-plane:/# kubectl exec -n airbyte-abctl deploy/airbyte-abctl-workload-launcher -- env | grep -E "JOB_MAIN|SIDECAR" JOB_MAIN_CONTAINER_MEMORY_LIMIT=4Gi JOB_MAIN_CONTAINER_CPU_LIMIT=3 JOB_MAIN_CONTAINER_MEMORY_REQUEST=2Gi JOB_MAIN_CONTAINER_CPU_REQUEST=1 CONNECTOR_SIDECAR_IMAGE=airbyte/connector-sidecar:1.8.5 root@airbyte-abctl-control-plane:/#C01AHCD885S
k
The env vars have been successfully updated on the workload-launcher! Your new instance-wide defaults are now: • 
JOB_MAIN_CONTAINER_MEMORY_LIMIT=4Gi
• 
JOB_MAIN_CONTAINER_CPU_LIMIT=3
• 
JOB_MAIN_CONTAINER_MEMORY_REQUEST=2Gi
• 
JOB_MAIN_CONTAINER_CPU_REQUEST=1
These match the values you saw in your original sync logs, which is a good baseline. [Configuring Connector Resources] Next steps: 1. Your large MySQL connection should still be using the higher limits (20Gi memory) you set directly in the 
connection
table — that connection-level setting takes highest precedence over these instance-wide env vars. 2. Trigger a new sync on your large MySQL connection and check the new pod logs to confirm it launches with the connection-level resource requirements (20Gi/13Gi memory), not the instance-wide 4Gi. 3. Monitor whether all rows are loaded this time. > ⚠️ Keep in mind the open bug where resource configurations may still be ignored in some 
abctl
deployments. [abctl resource limits issue] If the pod still shows 4Gi limits instead of your 20Gi connection-level override, that bug may be affecting you. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai how about the on going jobs?
k
Based on the knowledge sources available, there is no explicit information about how resource configuration changes affect already running/ongoing jobs. However, based on what is documented: • Resource requirements are applied when a replication pod is launched. The logs show resources are set at pod creation time (e.g., 
Launching replication pod: replication-job-XXXXX-attempt-0 with containers...
). [Configuring Connector Resources] This means: • Already running jobs will not be affected by the changes you just made — the pods for those jobs were already launched with the old resource values. • New jobs triggered after your changes (both the env var update and the connection-level 
resource_requirements
DB update) will pick up the new resource limits when their pods are created. If you have an ongoing sync that you suspect is failing due to resource constraints, you would need to cancel it and re-trigger it so a new pod is launched with the updated resource configuration. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by