https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I am trying to sync a connector wit...
# replication-ask-ai
h
@kapa.ai I am trying to sync a connector with S3 as the destination. After few tens of minutes the sync dies and I receive this error 
aws_last_error: AWS_ERROR_HTTP_CONNECTION_CLOSED(2058)
Copy code
"stacktrace" : "software.amazon.awssdk.crt.CrtRuntimeException: HttpClientConnection.MakeRequest: Unable to Execute Request (aws_last_error: AWS_ERROR_HTTP_CONNECTION_CLOSED(2058), The connection has closed or is closing.) AWS_ERROR_HTTP_CONNECTION_CLOSED(2058)\n\tat software.amazon.awssdk.crt.http.HttpClientConnection.httpClientConnectionMakeRequest(Native Method)\n\tat software.amazon.awssdk.crt.http.HttpClientConnection.makeRequest(HttpClientConnection.java:50)\n\tat aws.sdk.kotlin.crt.http.HttpClientConnectionJVM.makeRequest(HttpClientConnectionJVM.kt:17)\n\tat aws.smithy.kotlin.runtime.http.engine.crt.ConnectionManager$LeasedConnection.makeRequest(ConnectionManager.kt)\n\tat aws.smithy.kotlin.runtime.http.engine.crt.CrtHttpEngine.roundTrip(CrtHttpEngine.kt:77)\n\tat aws.smithy.kotlin.runtime.http.engine.crt.CrtHttpEngine$roundTrip$1.invokeSuspend(CrtHttpEngine.kt)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.internal.ScopeCoroutine.afterResume(Scopes.kt:35)\n\tat kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:101)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)\n\tat kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)\n",
  "stacktrace" : "io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 1\n\tat io.airbyte.container.orchestrator.worker.DestinationReader.run(ReplicationTask.kt:55)\n\tat io.airbyte.container.orchestrator.worker.DestinationReader$run$1.invokeSuspend(ReplicationTask.kt)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n\tSuppressed: io.airbyte.workers.internal.exception.DestinationException: Destination process message delivery failed\n\t\tat io.airbyte.container.orchestrator.worker.DestinationWriter.run(ReplicationTask.kt:99)\n\t\tat io.airbyte.container.orchestrator.worker.DestinationWriter$run$1.invokeSuspend(ReplicationTask.kt)\n\t\t... 5 more\n\tCaused by: java.io.IOException: Broken pipe\n\t\tat java.base/sun.nio.ch.UnixFileDispatcherImpl.write0(Native Method)\n\t\tat java.base/sun.nio.ch.UnixFileDispatcherImpl.write(UnixFileDispatcherImpl.java:65)\n\t\tat java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:137)\n\t\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:102)\n\t\tat java.base/sun.nio.ch.IOUtil.write(IOUtil.java:72)\n\t\tat java.base/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:300)\n\t\tat java.base/sun.nio.ch.ChannelOutputStream.writeFully(ChannelOutputStream.java:68)\n\t\tat java.base/sun.nio.ch.ChannelOutputStream.write(ChannelOutputStream.java:105)\n\t\tat java.base/sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:309)\n\t\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:381)\n\t\tat java.base/sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:357)\n\t\tat java.base/sun.nio.cs.StreamEncoder.lockedWrite(StreamEncoder.java:158)\n\t\tat java.base/sun.nio.cs.StreamEncoder.write(StreamEncoder.java:139)\n\t\tat java.base/java.io.OutputStreamWriter.write(OutputStreamWriter.java:219)\n\t\tat java.base/java.io.BufferedWriter.implFlushBuffer(BufferedWriter.java:178)\n\t\tat java.base/java.io.BufferedWriter.flushBuffer(BufferedWriter.java:163)\n\t\tat java.base/java.io.BufferedWriter.implWrite(BufferedWriter.java:334)\n\t\tat java.base/java.io.BufferedWriter.write(BufferedWriter.java:313)\n\t\tat java.base/java.io.Writer.write(Writer.java:278)\n\t\tat io.airbyte.container.orchestrator.worker.io.AirbyteMessageBufferedWriter.write(AirbyteMessageBufferedWriter.kt:29)\n\t\tat io.airbyte.container.orchestrator.worker.io.LocalContainerAirbyteDestination.acceptWithNoTimeoutMonitor(LocalContainerAirbyteDestination.kt:133)\n\t\tat io.airbyte.container.orchestrator.worker.io.LocalContainerAirbyteDestination.accept(LocalContainerAirbyteDestination.kt:92)\n\t\tat io.airbyte.container.orchestrator.worker.DestinationWriter.run(ReplicationTask.kt:97)\n\t\t... 6 more\n",
k
Based on the knowledge sources, the stack trace indicates that the Kotlin-based AWS CRT HTTP client engine used by the newer S3 destination connector (
destination-s3
/ 
destination-s3-data-lake
) is dropping the connection (
aws_last_error: AWS_ERROR_HTTP_CONNECTION_CLOSED(2058)
), which causes the destination process to crash and results in a 
Broken pipe
exception in the replication orchestrator [Github Issue (open): destination-s3-data-lake (Iceberg): Add explicit retry/backoff logic for AWS SDK operations; Github Issue (open): class io.airbyte.cdk.load.data.NullValue cannot be cast to class io.airbyte.cdk.load.data.TimestampWithoutTimezoneValue...]. Here are the primary causes and troubleshooting steps identified in the knowledge base: 1. Upgrade to the Latest Connector Version Earlier 
1.x
versions of the S3 connector (such as 
1.5.x
through 
1.9.1
) had recurring network/socket issues with the underlying Kotlin AWS CRT HTTP client (e.g., system call failures, socket timeouts, and dropped connections) [Github Issue (closed): [destination-s3] Connection test fails with Minio on v1.9.1...; Github Issue (open): [destination-s3] Fails with HttpException: System call failure in Airbyte OSS]. ā€¢ Update your S3 Destination connector to the latest version available (e.g., 
1.9.7
or later), as several fixes and improvements have been released for these CRT HTTP client issues [Github Issue (closed): [destination-s3] Connection test fails with Minio on v1.9.1...]. 2. Check MinIO / Custom S3 Endpoint Settings (If Applicable) If you are syncing to an S3-compatible service like MinIO or an on-premise endpoint behind a reverse proxy (such as Nginx): ā€¢ Path-Style Access: Certain versions had issues regarding path-style bucket access (resolved in version 
1.9.2+
) [Github Issue (closed): [destination-S3]does not work with Minio]. ā€¢ Proxy/Keepalive Timeouts: If connecting through a reverse proxy (e.g., Nginx), ensure 
keepalive
and long connection timeouts (e.g., 
proxy_connect_timeout 300;
, HTTP/1.1) are enabled, as long-running multi-part uploads can cause the proxy to prematurely close idle/streaming TCP connections [Github Issue (open): [destination-s3] Fails with HttpException: System call failure in Airbyte OSS; Github Issue (closed): [destination-S3]does not work with Minio]. 3. AWS SDK Retry Limitations Known issues in the S3 connector codebase note that the S3 client using the CRT HTTP engine currently lacks explicit retry and exponential backoff handling for certain transient network drops during long sync operations [Github Issue (open): destination-s3-data-lake (Iceberg): Add explicit retry/backoff logic for AWS SDK operations]. If transient network drops or proxy timeouts close the TCP socket mid-request, the connection fails with an unhandled 
AWS_ERROR_HTTP_CONNECTION_CLOSED
. Note: If you are already on the latest version of the destination connector and your network/proxy timeouts are sufficiently high, consider opening an issue on the Airbyte GitHub repository with your connector version and sync duration details [Github Issue (closed): [destination-s3] Connection test fails with Minio on v1.9.1...]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
h
@kapa.ai I'm using S3 destination version 1.9.7
k
Based on the knowledge sources, the latest release for the S3 destination connector is 
1.9.8
 [Documentation Integrations > Destinations > S3>; airbyte-integrations Connectors > Destination-s3 > metadata.yaml>]. Here are the recommended steps based on the available information: 1. Upgrade to Destination S3 1.9.8 Version 
1.9.8
updates the underlying CDK to version 
1.0.13
[Documentation Integrations > Destinations > S3>]. Upgrading may help resolve runtime engine issues related to socket connections and data transfers. 2. Inspect Custom S3 / MinIO Proxies & Timeouts (If Applicable) The underlying error 
AWS_ERROR_HTTP_CONNECTION_CLOSED(2058)
occurs when the server or an intermediate proxy closes the HTTP connection during long-running streaming/multi-part requests [Github Issue (closed): [destination-S3]does not work with Minio; Github Issue (closed): [destination-s3] Connection test fails with Minio on v1.9.1...]. If you are using MinIO, an on-premise endpoint, or a reverse proxy (e.g., Nginx): ā€¢ Ensure keepalive connections and higher read/write/proxy timeouts are configured. ā€¢ Ensure path-style access settings and network routes are stable for extended multi-part uploads [Github Issue (closed): [destination-S3]does not work with Minio]. If the error persists after upgrading to 
1.9.8
, the knowledge sources suggest opening or reopening an issue on the Airbyte GitHub repository with the sync duration and environment details [Github Issue (closed): [destination-s3] Connection test fails with Minio on v1.9.1...]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
5 Views
Open in Slack
PreviousNext
Powered by