https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I am waiting one hour for the sync ...
# replication-ask-ai
i
@kapa.ai I am waiting one hour for the sync to run and then after one hour it fails with 2025-12-24 121222 replication-orchestrator ERROR Error closing resource io.airbyte.container.orchestrator.worker.io.LocalContainerAirbyteDestination@1771d6a5; recording failure but continuing. 2025-12-24 121222 replication-orchestrator ERROR DestinationReader error: 2025-12-24 121222 replication-orchestrator INFO DestinationReader finished. 2025-12-24 121222 replication-orchestrator ERROR runJobs failed; recording failure but continuing to finish. then the enxt run says no records to load, but all my documents aare in airbyte_tmp_ collection rather than airbyte_raw, meaning they were lost in the process
k
It appears your sync is failing during the resource teardown phase, specifically when trying to close the destination connection. The error 
Error closing resource LocalContainerAirbyteDestination; recording failure but continuing
is a known log pattern that often occurs when the destination becomes unresponsive or fails to shut down gracefully [[S3 Data Lake Issue](https://github.com/airbytehq/airbyte/issues/63359); [Firestore Issue](https://github.com/airbytehq/airbyte/issues/70837)]. ### Why your data is in 
airbyte_tmp
When a sync fails, Airbyte does not store data in the final destination tables. Instead, the data remains in temporary tables (often prefixed with 
_airbyte_tmp_*
). Airbyte only moves data from these temporary tables to the final raw tables and deletes the temporary ones after a sync succeeds [[Failed Sync Forum](https://discuss.airbyte.io/t/2067)]. The knowledge sources state that if a sync fails, these temporary tables are not automatically deleted, which explains why you see your documents there but not in the final 
airbyte_raw
tables [[Database Error Forum](https://discuss.airbyte.io/t/3693); [Sync Failure Forum](https://discuss.airbyte.io/t/77)]. ### Potential Causes and Solutions 1. Heartbeat/Timeout Settings: Your sync is failing after exactly one hour, which might suggest a timeout. While the default destination timeout is 24 hours, you can check your 
flags.yaml
for the following configuration: * `destination-timeout-max-seconds`: The maximum time allowed for a call to the destination. * `heartbeat-max-seconds-between-messages`: Defaults to 3 hours for the source, but if your platform is configured differently, it might be killing the process [[Heartbeats 2.0](https://docs.airbyte.com/platform/2.0/understanding-airbyte/heartbeats#technical-details)]. 2. Resource Constraints: One-hour failures are sometimes related to the environment killing the process (e.g., OOM or network timeouts). You may need to tweak timeout variables in your operator guides [[Worker Cancellation](https://discuss.airbyte.io/t/3171)]. 3. Manual Cleanup: Because Airbyte currently does not automatically clean up these temporary tables on failure, you may need to manually delete them in your destination database to free up space or clear the state before attempting another sync [[Failed Sync Forum](https://discuss.airbyte.io/t/2067)]. If the next run says "no records to load," it suggests that the source state was updated despite the destination failure. You may need to reset your connection to re-sync the data that is currently stuck in the temporary tables.
i
@kapa.ai I see the ammory for destination burstet to 1.5 but The limits were still 2gb, what could cause this ?
k
Sorry @Iulian, it looks like your question contains files, which I cannot process. Please reformulate your question without including any files.
i
I seea pattern occuring, the destination containers, suddenly after a long time, burts in memory, that's ehat causes it to fail, why do they burst like that ?
@kapa.ai I see a pattern occuring, the destination containers, suddenly after a long time, burts in memory, that's ehat causes it to fail, why do they burst like that ?
k
A message from kapa.ai
Based on the provided knowledge sources, there are several documented reasons why Airbyte destination containers may experience a sudden "burst" or buildup in memory usage that eventually leads to a crash (OOM): * Batch Accumulation and Record Size: Destination connectors generally read records into a buffer until they reach a specific batch size—currently set at 10,000 records—before flushing them to the destination. If the table has large row sizes (e.g., several MBs per row), this 10,000-record buffer can grow to several gigabytes, causing a sudden memory spike [[How memory request and limit works](https://discuss.airbyte.io/t/2214); [Mysql to clickhouse fails](https://discuss.airbyte.io/t/3401)]. * In-Memory Buffering vs. Streaming: Certain destinations (like S3 Data Lake or Iceberg) may have issues where the connector buffers data in memory instead of streaming it directly to the target. This leads to a linear climb in memory usage until the container hits its limit and is killed [[S3 Data Lake OOMKilled](https://github.com/airbytehq/airbyte/issues/70247)]. * Java Memory Behavior: Many Airbyte connectors are Java-based and follow default JVM behavior. By default, they may only use up to 1/4 of the host’s allocated memory. If the container reaches this internal limit, it will throw an 
OutOfMemoryError
even if the host has more RAM available [[Mysql to clickhouse fails](https://discuss.airbyte.io/t/3401)]. * Memory Leaks in Resource Teardown: There are reports of memory leaks where resources (like S3 file objects or internal buffers) are not closed properly during the sync. This causes memory to build up over time until it reaches a breaking point, often manifesting as a crash after a long period of operation [[S3 Data Lake OOMKilled](https://github.com/airbytehq/airbyte/issues/70247); [Shopify connector memory build up](https://discuss.airbyte.io/t/3553)]. * Final Copy Operations: In some connectors (like MongoDB), the destination writes to a temporary collection first. The final step—moving data from temporary to permanent collections—can sometimes trigger a memory exception if the operation is handled entirely in memory rather than as a database-side command [[destination-mongodb Out of memory](https://github.com/airbytehq/airbyte/issues/48851)]. To debug this, you can check your job logs for the 
ResourceRequirements
line to confirm the memory limits being applied to your destination container [[Configuring Connector Resources](https://docs.airbyte.com/platform/2.0/operator-guides/configuring-connector-resources#debugging)].
Open in Slack
PreviousNext
Powered by