hey guys, i am facing this issue with my k8s airbyte deployment, looks like a connection is stuck and spamming retries, i can see these logs

2022-12-06T16:53:37.423923573Z stdout F {"level":"warn","ts":"2022-12-06T16:53:37.423Z","msg":"history size exceeds warn limit.","shard-id":4,"address":"10.1.1.248:7234","component":"history-cache","wf-namespace-id":"769c5c50-2fb9-4b30-a8d3-ea88dd398f4c","wf-id":"connection_manager_84a09e16-67ec-432b-9cc3-1224516d6ad2","wf-run-id":"a3dd8fcd-19b6-4fc9-8faf-b7c3075bf0c0","wf-history-size":32381433,"wf-event-count":4380,"logging-call-at":"context.go:920"}

2022-12-06T16:52:07.390203639Z stdout F java.lang.IllegalStateException: Transitioning Job 1881 from JobStatus SUCCEEDED to INCOMPLETE is not allowed. The only valid statuses that an be transitioned to from SUCCEEDED are []
2022-12-06T18:52:07+02:00
2022-12-06T16:52:07.390170567Z stdout F 2022-12-06 16:52:07 
WARN
 i.t.i.a.ActivityTaskExecutors$BaseActivityTaskExecutor(execute):114 - Activity failure. ActivityId=d551e5c8-e3ea-336d-82e8-c03469ea67b7, activityType=AttemptFailureWithAttemptNumber, attempt=1
2022-12-06T18:51:49+02:00
2022-12-06T16:51:49.228566415Z stdout F 2022-12-06 16:51:49 
INFO
 i.a.w.t.s.ConnectionManagerWorkflowImpl(runMandatoryActivityWithOutput):628 - Waiting PT10M before restarting the workflow for connection 1acb8818-fdba-4780-840e-d552bd2d4684, to prevent spamming temporal with restarts.

theses errors keep showing in the logs, and i can’t delete the connection for some reason too, any help ?

Hi Rami, I need some more details - Are you using helm or kustomize? Did you upgrade/do something else right before this started happening? Could you include the full logs? Thanks!

i am using helm, no just normal usage,

OK, let me look into this - I know people run into issues with helm quite a bit!