https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# replication-ask-ai
s
This message was deleted.
k
Thinking completed MySQL Initial Sync PK State Log Explanation The log lines you're seeing are normal and expected behavior during a MySQL initial sync (full table load) using primary key-based checkpointing. What's happening Line 1 — State checkpoint emission:
Copy code
Emitting initial sync pk state for stream pd_microsoft_PD_TM_STATISTICS_FUZZY_BANDS,
state is PrimaryKeyLoadStatus[version=2, stateType=primary_key,
pkName=TM_STATISTICS_ID, pkVal=3747789, incrementalState={}, additionalProperties={}]
Airbyte's MySQL source is performing an initial load of the stream 
pd_microsoft_PD_TM_STATISTICS_FUZZY_BANDS
. During this phase, it reads the table in primary key order and periodically emits state checkpoint messages to track progress. The state contains: • 
stateType=primary_key
 — confirms this is a PK-based initial load (not cursor-based) • 
pkName=TM_STATISTICS_ID
 — the primary key field being used as the watermark • 
pkVal=3747789
 — the last primary key value processed so far (the watermark) • 
incrementalState={}
 — the incremental (CDC) state to switch to once the initial load completes; it's empty here, meaning it will be populated when the snapshot finishes This checkpoint mechanism ensures that if the sync fails, the next run can resume from 
TM_STATISTICS_ID = 3747789
 rather than starting over from the beginning. [State & Checkpointing] Line 2 — Progress indicator:
Copy code
Records read: 4525000 (1 GB)
The replication orchestrator has read 4,525,000 records (~1 GB) from the source so far. This is a large initial load. Key takeaway These are informational logs, not errors. The sync is progressing normally. The state message at 
pkVal=3747789
will only be passed back to the source on the next run if it is also successfully committed by the destination — ensuring no data loss even if the sync fails mid-way. [State & Checkpointing] If you're seeing performance concerns with a 1 GB+ initial load, that is expected for large tables during the first sync. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai THAT INITIAL SYNC IS TAKING TOO MUCH TIME AND ALSO AIRBYTE FINISH THE JOB BEFORE LOAD ALL TABLE THAN RESUME THE INITIAL SYNC IN ANOTHER JOB
k
Thinking completed MySQL Initial Sync: Slow Performance & Resume Behavior Resume Behavior is Expected What you're observing — the sync finishing a job and resuming the initial load in a subsequent job — is intentional and by design. Airbyte's MySQL source uses primary key-based checkpointing to paginate through the table in chunks. If the sync stops (for any reason), the next run picks up from the last committed 
pkVal
rather than restarting from scratch. [Resumability] Why the Initial Sync is Slow For large tables, the initial snapshot can take a very long time. A few factors contribute to this: • Table size — your logs already show 4,525,000 records (~1 GB) read in one attempt, suggesting the full table is significantly larger. • Resource allocation — CPU and RAM assigned to the source/destination containers directly impact throughput. • Query timeouts — MySQL's 
max_execution_time
setting can interrupt queries mid-sync, forcing retries. What You Can Do 1. Check for Query Timeouts If you see this error in your logs:
Copy code
MySQL Query Timeout: The sync was aborted because the query took too long to return results, will retry.
You should increase MySQL's 
max_execution_time
. Go to your Source Configuration → Optional Fields → Checkpoint Target Time Interval (default: 300s / 5 minutes), then set 
max_execution_time
to at least that value in milliseconds (e.g., 
300000
). [MySQL Troubleshooting] 2. Take Advantage of Faster Sync Speed Airbyte has a faster sync speed feature that can make syncs 4–10x faster for supported connectors. MySQL → S3/BigQuery/ClickHouse/Azure Blob Storage are supported. To use it: • Upgrade to the latest MySQL source connector version • If self-managed, upgrade to Airbyte version 1.8 or later • Cloud users get this automatically Benchmarks show a 170 GB MySQL → S3 sync going from 1h 35m down to 22 minutes on C4 nodes. [Faster Sync Speed] 3. Review Infrastructure Resources If self-managing, consider using higher-performance CPU nodes (e.g., C4 on GCP). CPU performance is now directly correlated with sync speed and efficiency. [Faster Sync Speed] Summary 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
r
@kapa.ai SO MAYBE THE problem is the resources alocated:
Copy code
2026-10-21 10:04:52 platform INFO [initContainer] image: airbyte/workload-init-container:2.1.0 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=2}, requests={memory=2Gi, cpu=2}, additionalProperties={})
2026-10-21 10:04:52 platform INFO Launching replication pod: replication-job-52678-attempt-0 (selectors = {}) with containers:
2026-10-21 10:04:52 platform INFO [source] image: airbyte/source-mysql:3.1.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=2}, requests={memory=1Gi, cpu=1}, additionalProperties={})
2026-10-21 10:04:52 platform INFO [destination] image: airbyte/destination-snowflake:5.0.1 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=2}, requests={memory=2Gi, cpu=1}, additionalProperties={})
2026-10-21 10:04:52 platform INFO [orchestrator] image: airbyte/container-orchestrator:2.1.0 resources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=2}, requests={memory=2Gi, cpu=2}, additionalProperties={})
k
Thinking completed Resource Allocation Analysis Looking at your logs, the resources allocated are: The Problem You are running Airbyte 2.1.0, which has a known bug where 
REPLICATION_ORCHESTRATOR_*
and related resource configurations are ignored — the orchestrator was hardcoded to use default values regardless of what you configured. [GitHub Issue] Additionally, your source container is only getting: • 
1 CPU
request / 
2 CPU
limit • 
1Gi
memory request / 
2Gi
memory limit For a large MySQL initial sync, this is likely a bottleneck. The Fix: Upgrade to Chart 2.2.0 The resource configuration bug was fixed in Helm chart 2.2.0. After upgrading, you can properly configure resources: [GitHub Issue]
Copy code
global:
  workloads:
    resources:
      useConnectorResourceDefaults: false
      mainContainer:       # source & destination containers
        cpu:
          request: "2"
          limit: "4"
        memory:
          request: "2Gi"
          limit: "4Gi"
      replication:         # orchestrator container
        cpu:
          request: "1"
          limit: "2"
        memory:
          request: "2Gi"
          limit: "4Gi"
Copy code
helm repo add airbyte-v2 https://airbytehq.github.io/charts
helm repo update
helm upgrade --install airbyte airbyte-v2/airbyte --version 2.2.0 -n airbyte -f values.yaml
Also Consider • Faster sync speed is available for MySQL → Snowflake is not currently listed as a supported destination for the faster sync feature. Supported destinations are S3, BigQuery, ClickHouse, and Azure Blob Storage. [Faster Sync Speed] • Upgrading to 2.2.0 is the most impactful immediate step to ensure your resource settings are actually applied. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
r
@kapa.ai getting this erro at the end: 2026-10-20 183922 replication-orchestrator ERROR Destination has timed out on notifyEndOfInput call 2026-10-20 183922 replication-orchestrator WARN Destination has timed out; failing the workload. 2026-10-20 183922 replication-orchestrator WARN airbyte-destination gobbler IOException: null. Typically happens when cancelling a job. 2026-10-20 183922 replication-orchestrator ERROR DestinationReader error: 2026-10-20 183922 replication-orchestrator INFO DestinationReader finished. 2026-10-20 183922 replication-orchestrator ERROR SourceReader error: 2026-10-20 183922 replication-orchestrator INFO MessageProcessor finished. 2026-10-20 183922 replication-orchestrator INFO SourceReader finished. 2026-10-20 184022 replication-orchestrator WARN Failed to wait for exit value file /dest/exitCode.txt to be found. 2026-10-20 184022 replication-orchestrator WARN Error sending heartbeat 2026-10-21 094314 replication-orchestrator WARN Workload in terminal state; exiting.
k
Thinking completed Destination Timeout Error Analysis What's Happening The error sequence shows: 1. 
Destination has timed out on notifyEndOfInput call
 — The destination (Snowflake) took too long to finish processing and flushing all records after the source finished sending data. 2. 
Failed to wait for exit value file /dest/exitCode.txt
 — Because the destination timed out and was killed, it never wrote its exit code file, causing the orchestrator to fail. [GitHub Issue] 3. 
Workload in terminal state; exiting
 — The orchestrator eventually gave up (~15 hours later). This is directly related to the resource constraints we identified earlier. With only 2Gi memory / 2 CPU for the destination Snowflake container, it cannot flush large batches fast enough before the timeout is hit. Root Causes Recommended Fixes 1. Upgrade to Helm Chart 2.2.0 (Most Important) As discussed, the resource configuration bug in 2.1.0 means your settings may not be applied. After upgrading, increase destination resources:
Copy code
global:
  workloads:
    resources:
      useConnectorResourceDefaults: false
      mainContainer:
        cpu:
          request: "2"
          limit: "4"
        memory:
          request: "4Gi"
          limit: "8Gi"
2. Upgrade the MySQL Source Connector You are running source-mysql:3.1.1, which is quite old. Newer versions have improved checkpointing and state management that reduce the amount of data buffered at end-of-sync. Check the latest version in the Airbyte connector registry. 3. Upgrade the Snowflake Destination Connector You are running destination-snowflake:5.0.1. Newer versions have improved batching and flushing behavior that can reduce the time spent on 
notifyEndOfInput
. Summary Flow of the Failure
Copy code
Source finishes sending records
        ↓
Destination starts flushing batch to Snowflake
        ↓
Flush takes too long (under-resourced container)
        ↓
notifyEndOfInput timeout hit
        ↓
Destination killed → no exitCode.txt written
        ↓
Orchestrator fails → sync resumes from last PK checkpoint on next run
The good news is that because of PK-based checkpointing, the next sync run will resume from where it left off rather than restarting from scratch. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
Open in Slack
PreviousNext
Powered by