Thomas Le Naour
05/30/2024, 12:42 PMio.temporal.failure.ActivityFailure: Activity with activityType='RunWithWorkload' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=f1f21bc0-4183-370f-b174-8df6b69c622b, identity='1@prod-worker-6556947fdb-r2bhv', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
at java.base/java.lang.Thread.getStackTrace(Thread.java:2450)
at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:49)
at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83)
at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60)
at jdk.proxy3/jdk.proxy3.$Proxy141.runWithWorkload(Unknown Source)
at io.airbyte.workers.temporal.check.connection.CheckConnectionWorkflowImpl.run(CheckConnectionWorkflowImpl.java:53)
at CheckConnectionWorkflowImplProxy.run$accessor$Occdnbed(Unknown Source)
at CheckConnectionWorkflowImplProxy$auxiliary$Smun3I7I.call(Unknown Source)
at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79)
at CheckConnectionWorkflowImplProxy.run(Unknown Source)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339)
at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314)
at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70)
at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135)
at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102)
at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107)
at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
kapa.ai
05/30/2024, 12:42 PMCheckConnectionWorkflowImpl
. This might be due to an internal error in the scheduler. Unfortunately, the logs do not provide a specific cause. You might want to check the Airbyte issue tracker for any similar issues or updates. If the above steps do not resolve the issue, you might want to consider reaching out to the Airbyte community or support for further assistance. You can also check the [Airbyte GitHub issues page](https://github.com/airbytehq/airbyte/issues) for any ongoing or resolved issues that might be related. If you need more detailed assistance, please provide additional context or logs, and I can help you further.Thomas Le Naour
05/30/2024, 12:49 PMretryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
?Thomas Le Naour
05/30/2024, 12:50 PMretryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
?kapa.ai
05/30/2024, 12:50 PMretryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
error indicates that the system has attempted to perform a specific activity multiple times but has failed each time, reaching the maximum number of retry attempts allowed. This error can occur in various contexts within Airbyte, as seen in the provided knowledge sources.
### Contexts and Examples
1. Destination-Snowflake Connector Issue:
- Users have reported this error when configuring or running the destination-snowflake connector after upgrading Airbyte to a newer version. The error appears consistently across all existing pipelines, saved connections, and new connections.
- Example log output:
shell
io.temporal.failure.ActivityFailure: Activity with activityType='RunWithJobOutput' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=1477922c-d4ea-3691-9381-3ee09744c8d1, identity='1@9a778d6165cc', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
- [Source](https://github.com/airbytehq/airbyte/issues/36401)
2. MySQL Normalization Failure:
- This error also appears in the context of MySQL normalization, where the normalization process fails due to an import error in the Pydantic library.
- Example log output:
shell
io.temporal.failure.ActivityFailure: scheduledEventId=24, startedEventId=25, activityType='Normalize', activityId='788a80eb-aacb-3fb2-b9b3-bb6883c0cf8d', identity='1@2ae255f3e053', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
- [Source](https://github.com/airbytehq/airbyte/discussions/33748)
3. General Workflow Execution Failures:
- The error can also occur in general workflow execution scenarios, where a connection manager workflow fails to run an activity and is scheduled to restart after a delay.
- Example log output:
shell
io.temporal.failure.ActivityFailure: scheduledEventId=211, startedEventId=212, activityType='AttemptFailureWithAttemptNumber', activityId='597786c9-dc03-30a8-b81d-0255bc859e04', identity='1@a036e1485fbd', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
- [Source](https://github.com/airbytehq/airbyte/discussions/28859)
### Possible Causes and Solutions
- Breaking Changes in Airbyte Versions:
- Upgrading Airbyte to a newer version may introduce breaking changes that affect the connectors. For instance, users have reported issues after upgrading to versions like v0.51.0 and v0.55.0.
- Solution: Review the release notes for the Airbyte version you upgraded to and check for any breaking changes or required configuration updates.
- Connector-Specific Issues:
- The error might be specific to certain connectors, such as the destination-snowflake connector.
- Solution: Try downgrading the connector to a previous stable version or check for any updates or patches that address the issue.kapa.ai
05/30/2024, 12:50 PMretryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
error indicates that the system has attempted to perform a specific activity multiple times but has failed each time, reaching the maximum number of retry attempts allowed. This error can occur in various contexts within Airbyte, as seen in the provided knowledge sources. ### Contexts and Examples 1. Destination-Snowflake Connector Issue: - Users have reported this error when configuring or running the destination-snowflake connector after upgrading Airbyte to a newer version. The error appears consistently across all existing pipelines, saved connections, and new connections. - Example log output: shell io.temporal.failure.ActivityFailure: Activity with activityType='RunWithJobOutput' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=1477922c-d4ea-3691-9381-3ee09744c8d1, identity='1@9a778d6165cc', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
- [Source](https://github.com/airbytehq/airbyte/issues/36401) 2. MySQL Normalization Failure: - This error also appears in the context of MySQL normalization, where the normalization process fails due to an import error in the Pydantic library. - Example log output: shell io.temporal.failure.ActivityFailure: scheduledEventId=24, startedEventId=25, activityType='Normalize', activityId='788a80eb-aacb-3fb2-b9b3-bb6883c0cf8d', identity='1@2ae255f3e053', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
- [Source](https://github.com/airbytehq/airbyte/discussions/33748) 3. General Workflow Execution Failures: - The error can also occur in general workflow execution scenarios, where a connection manager workflow fails to run an activity and is scheduled to restart after a delay. - Example log output: shell io.temporal.failure.ActivityFailure: scheduledEventId=211, startedEventId=212, activityType='AttemptFailureWithAttemptNumber', activityId='597786c9-dc03-30a8-b81d-0255bc859e04', identity='1@a036e1485fbd', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
- [Source](https://github.com/airbytehq/airbyte/discussions/28859) ### Possible Causes and Solutions - Breaking Changes in Airbyte Versions: - Upgrading Airbyte to a newer version may introduce breaking changes that affect the connectors. For instance, users have reported issues after upgrading to versions like v0.51.0 and v0.55.0. - Solution: Review the release notes for the Airbyte version you upgraded to and check for any breaking changes or required configuration updates. - Connector-Specific Issues: - The error might be specific to certain connectors, such as the destination-snowflake connector. - Solution: Try downgrading the connector to a previous stable version or check for any updates or patches that address the issue. - Internal Scheduler Errors: - The error message indicates an internal error in the scheduler used by Airbyte. - Solution: Investigate the scheduler logs for more details and consider reaching out to Airbyte support or the community for assistance. ### Conclusion The RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
error is a common