https://linen.dev logo
Join Slack
Powered by
Im having this error whili moving data from snowfl...
# replication-ask-ai
g
Im having this error whili moving data from snowflake source to snowflake destiny: Activity with activityType='RunWithJobOutput' failed: 'Activity task timed out'. scheduledEventId=12, startedEventId=13, activityId=c6684ded-c3c4-3b8b-8767-83c7af936d7d, identity='', retryState=RETRY_STATE_NON_RETRYABLE_FAILURE
k
It looks like you're encountering a timeout error while moving data from a Snowflake source to a Snowflake destination. The error message indicates that the activity task timed out and reached a non-retryable failure state. Here are some steps you can take to troubleshoot and potentially resolve this issue: 1. Check Network Connectivity: Ensure that there are no network issues between your Airbyte instance and the Snowflake databases. Network latency or interruptions can cause timeouts. 2. Review Resource Limits: Verify that your Snowflake account has sufficient resources (e.g., compute credits, warehouse size) to handle the data transfer. Insufficient resources can lead to timeouts. 3. Increase Timeout Settings: If possible, increase the timeout settings for the connection or the specific activity in Airbyte. This can help accommodate longer-running tasks. 4. Check Airbyte Logs: Review the Airbyte logs for more detailed information about the failure. The logs can provide insights into what might be causing the timeout. 5. Update Connectors: Ensure that you are using the latest versions of the Snowflake source and destination connectors. Sometimes, updating to the latest version can resolve known issues. 6. Retry the Operation: Sometimes, transient issues can cause timeouts. Retry the operation to see if it succeeds on a subsequent attempt. If the issue persists, you may want to open a new topic on the [Airbyte forum](https://discuss.airbyte.io) or check existing issues on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte/issues) for similar problems and potential solutions. For reference, here is a similar issue reported on the Airbyte GitHub repository: - [Source S3: Caused by: io.temporal.failure.TimeoutFailure: message='activity timeout'](https://github.com/airbytehq/airbyte/issues/25955) If you need further assistance, please provide more details about your setup and any relevant logs. I hope this helps! If you have any more questions, feel free to ask.
g
@kapa.ai what does this error mean: java.lang.RuntimeException: Failed to upload buffer to stage and commit to destination
k
The error message 
java.lang.RuntimeException: Failed to upload buffer to stage and commit to destination
indicates that there was a failure during the process of uploading data to a staging area and subsequently committing that data to the final destination. This error can occur in various contexts within Airbyte, particularly when dealing with different destination connectors. Here are some specific instances and causes of this error from the provided knowledge sources: 1. BigQuery Destination: - The error can occur during the initial sync for a CDC (Change Data Capture) connection from a Postgres source to a BigQuery destination. The failure happens while uploading staging files to the destination table. - Example stack trace:
Copy code
java
     java.lang.RuntimeException: Failed to upload buffer to stage and commit to destination
         at io.airbyte.integrations.destination.bigquery.BigQueryStagingConsumerFactory.lambda$flushBufferFunction$4(BigQueryStagingConsumerFactory.java:169)
         ...
     Caused by: java.lang.RuntimeException: [JobId{project=GCP_PROJECT, job=5196eeb4-5821-4682-a3c6-6b51dd0a64bf, location=US}] Failed to upload staging files to destination table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=source, tableId=_airbyte_raw_adapter_line_items}} (source)
         at io.airbyte.integrations.destination.bigquery.BigQueryGcsOperations.lambda$copyIntoTableFromStage$0(BigQueryGcsOperations.java:156)
         ...
- [Source](https://github.com/airbytehq/airbyte/issues/26270) 2. Snowflake Destination: - The error can also occur when syncing tables with the same name from different sources in parallel to a Snowflake destination. The failure is due to exceptions thrown while uploading records into the stage. - Example stack trace:
Copy code
java
     java.lang.RuntimeException: Failed to upload buffer to stage and commit to destination
         at io.airbyte.integrations.destination.staging.AsyncFlush.flush(AsyncFlush.java:126)
         ...
     Caused by: java.lang.RuntimeException: Exceptions thrown while uploading records into stage: net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error:
     Stage 'AIRBYTE_SYNCS_FULL_REFRESH."airbyte_internal"."companies"' does not exist or not authorized.
         at io.airbyte.integrations.destination.snowflake.SnowflakeInternalStagingSqlOperations.uploadRecordsToStage(SnowflakeInternalStagingSqlOperations.java:94)
         ...
- [Source](https://github.com/airbytehq/airbyte/issues/30531)
The error message 
java.lang.RuntimeException: Failed to upload buffer to stage and commit to destination
indicates that there was a failure during the process of uploading data to a staging area and subsequently committing that data to the final destination. This error can occur in various contexts within Airbyte, particularly when dealing with different destination connectors. Here are some specific instances and causes of this error from the provided knowledge sources: 1. BigQuery Destination: - The error can occur during the initial sync for a CDC (Change Data Capture) connection from a Postgres source to a BigQuery destination. The failure happens while uploading staging files to the destination table. - Example stack trace:
Copy code
java      java.lang.RuntimeException: Failed to upload buffer to stage and commit to destination          at io.airbyte.integrations.destination.bigquery.BigQueryStagingConsumerFactory.lambda$flushBufferFunction$4(BigQueryStagingConsumerFactory.java:169)          ...      Caused by: java.lang.RuntimeException: [JobId{project=GCP_PROJECT, job=5196eeb4-5821-4682-a3c6-6b51dd0a64bf, location=US}] Failed to upload staging files to destination table GenericData{classInfo=[datasetId, projectId, tableId], {datasetId=source, tableId=_airbyte_raw_adapter_line_items}} (source)          at io.airbyte.integrations.destination.bigquery.BigQueryGcsOperations.lambda$copyIntoTableFromStage$0(BigQueryGcsOperations.java:156)          ...
- [Source](https://github.com/airbytehq/airbyte/issues/26270) 2. Snowflake Destination: - The error can also occur when syncing tables with the same name from different sources in parallel to a Snowflake destination. The failure is due to exceptions thrown while uploading records into the stage. - Example stack trace:
Copy code
java      java.lang.RuntimeException: Failed to upload buffer to stage and commit to destination          at io.airbyte.integrations.destination.staging.AsyncFlush.flush(AsyncFlush.java:126)          ...      Caused by: java.lang.RuntimeException: Exceptions thrown while uploading records into stage: net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error:      Stage 'AIRBYTE_SYNCS_FULL_REFRESH."airbyte_internal"."companies"' does not exist or not authorized.          at io.airbyte.integrations.destination.snowflake.SnowflakeInternalStagingSqlOperations.uploadRecordsToStage(SnowflakeInternalStagingSqlOperations.java:94)          ...
- [Source](https://github.com/airbytehq/airbyte/issues/30531) 3. Redshift Destination: - The error can occur in the context of a Redshift destination due to a 
StackOverflowError
in the JOOQ library. - Example stack trace:
Copy code
java      java.lang.RuntimeException: Failed to upload data from stage sor_hubspot_raw__stream_ab_contacts/2023_12_22_17_a23c67a1-3f97-49da-95ac-254046ab3db4/          at io.airbyte.cdk.integrations.destination.staging.GeneralStagingFunctions.copyIntoTableFromStage(GeneralStagingFunctions.java:101)          ...      Caused by: java.util.concurrent.CompletionException: java.lang.StackOverflowError          at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)          ...      Caused by: java.lang.StackOverflowError          at org.jooq.impl.Function.toSQLFunctionName(Function.java:634)          ...
- [Source](https://
Open in Slack
PreviousNext
Powered by