Hi team, need help figuring out why an initial sync using the snowflake source (to the S3 destination) is resulting in a network timeout. Incremental syncs work just fine! This appears to be a cold-start issue and can’t figure out if the issue is on the Airbyte EC2 instance or on the Snowflake side:

at io.airbyte.integrations.source.snowflake.SnowflakeSource.main(SnowflakeSource.java:47) Caused by: net.snowflake.client.jdbc.SnowflakeSQLLoggedException: JDBC driver internal error: Timeout waiting for the download of #chunk164(Total chunks: 1147) retry=1.

The sync payload size is around 20GB. The EC2 instance was initially an

xlarge

and got upgraded to a

2xlarge

, but doesn’t look like changing the instance type is helping. Any help appreciated! Airbyte Version 0.40.23

The error is from the underlying Snowflake JDBC driver, where the SnowflakeChunkDownloader does an async download of the result chunks from Snowflake. From your log it looks like there are too many chunks to download. Are there any other errors from the driver

@Vivek Ayer can you share the complete log file? Also could you try a sample table first to make sure the sync works?

Thanks folks! Standby for the log file

@Vivek Ayer I need the complete log file not only a snippet.

As mentioned earlier, its just too much data and the downloader is choking resulting in a timeout #chunk860 (Total chunks: 1161)

can you confirm you’re using latest version for both connectors? It work with a smaller table?

I’m going the share the full log file shortly

Usually users split large tables into single connections to have better control of their failures.

@Marcos Marx (Airbyte) connector versions: S3 (0.3.17); Snowflake (0.1.26) [both latest]

https://community.snowflake.com/s/article/JDBC-OutOfDirectMemoryError Vivek could you split the streams from source snowflake into individual connectinons and test again?

@Marcos Marx (Airbyte) kind of confused here, we only had one stream enabled, looks like the initial sync failed set to

Incremental | Append

. Subsequent syncs have been succeeding but they only picked up data after the cursor advanced

@Vivek Ayer incremental will read only new data records based on cursor field. not sure if option 3 will solve or impact in the issue.

@Marcos Marx (Airbyte) okay what are my options?

@Vivek Ayer did you find a solution for this?

not yet @Nohelia Merino!