@kapa.ai Just upgraded from 0.50.34 to 0.61.0 And now all connectors and destinations return io.temporal.failure.ActivityFailure: Activity with activityType='RunWithJobOutput' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=267ef4c7-61c0-3577-b806-2ddebf21b8d9, identity='1@7e46103c1cad', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED at java.base/java.lang.Thread.getStackTrace(Thread.java:2450) at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:49) at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83) at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60) at jdk.proxy2/jdk.proxy2.$Proxy119.runWithJobOutput(Unknown Source) at io.airbyte.workers.temporal.check.connection.CheckConnectionWorkflowImpl.run(CheckConnectionWorkflowImpl.java:55) at CheckConnectionWorkflowImplProxy.run$accessor$WNI1Y9oa(Unknown Source) at CheckConnectionWorkflowImplProxy$auxiliary$OLmflgf6.call(Unknown Source) at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) at CheckConnectionWorkflowImplProxy.run(Unknown Source) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339) at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314) at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70) at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135) at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102) at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107) at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)

It looks like you're encountering an issue with Temporal activities failing after upgrading from Airbyte version 0.50.34 to 0.61.0. The error message indicates that the activity

RunWithJobOutput

failed and reached the maximum retry attempts. Here are some steps you can take to troubleshoot and potentially resolve this issue: 1. Check Logs for More Details: - Review the logs for more detailed error messages that might indicate why the

RunWithJobOutput

activity is failing. Look for any specific exceptions or stack traces that could provide more context. 2. Verify Configuration: - Ensure that your configuration files and environment variables are correctly set up after the upgrade. Sometimes, configuration changes between versions can cause issues. 3. Check for Known Issues: - Review the [Airbyte GitHub issues](https://github.com/airbytehq/airbyte/issues) to see if there are any known issues related to the version you upgraded to. For example, there have been issues reported with Temporal activities in the past: - [Issue #33279](https://github.com/airbytehq/airbyte/issues/33279): Environment instability after upgrade. - [Issue #26673](https://github.com/airbytehq/airbyte/issues/26673): Infra error for

CheckConnectionWorkflow

. 4. Temporal SDK Compatibility: - Ensure that the Temporal SDK version you are using is compatible with Airbyte 0.61.0. There might be changes in the Temporal SDK that could affect how activities are executed. 5. Retry Mechanism: - Investigate the retry mechanism for the failing activity. The error message indicates that the retry state is

RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED

. You might need to adjust the retry policy or investigate why the activity is failing repeatedly. 6. Community and Support: - If you are unable to resolve the issue, consider reaching out to the Airbyte community for support. You can post on the [Airbyte forum](https://discuss.airbyte.io/) or open a new issue on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte/issues). Here is an example of a similar issue reported on GitHub:

plaintext io.temporal.failure.ActivityFailure: Activity with activityType='RunWithJobOutput' failed: 'Activity task failed'. scheduledEventId=12, startedEventId=13, activityId=267ef4c7-61c0-3577-b806-2ddebf21b8d9, identity='1@7e46103c1cad', retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED

For more details, you can refer to the following issues: - [Issue #33279](https://github.com/airbytehq/airbyte/issues/33279) - [Issue #26673](https://github.com/airbytehq/airbyte/issues/26673) If you need further assistance, please provide more details about the specific errors or logs you are seeing, and the community or support team can help you troubleshoot further.

Given Caused by: io.temporal.failure.ApplicationFailure: message='/tmp/workspace/9995a0a3-3ca9-4cf6-9ac0-d64fb8eac7e7/0', type='io.airbyte.workers.exception.WorkerException', nonRetryable=false at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:191) at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:149) at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:71) ... 18 more Caused by: io.temporal.failure.ApplicationFailure: message='/tmp/workspace/9995a0a3-3ca9-4cf6-9ac0-d64fb8eac7e7/0', type='java.nio.file.NoSuchFileException', nonRetryable=false at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:438) at java.base/java.nio.file.Files.createDirectory(Files.java:699) at io.airbyte.workers.process.DockerProcessFactory.create(DockerProcessFactory.java:121) ... 20 more This feels like file path issues.

A message from kapa.ai