I ve recently moved our airbyte deployment from abctl to a k Airbyte #ask-community-for-troubleshooting

I've recently moved our airbyte deployment from ab...

Zack Mattor

10/30/2025, 2:26 PM

I've recently moved our airbyte deployment from abctl to a k8s cluster and moved the database into a cloud sql managed database in GCP... all seems to have loaded and migrated fine besides all "jobs" or "workflows" seem to fail.... For example I can't perform a "test" on any destinations or sources... it just hangs forever, and I see no indication in any of the service's logs as to what's going on.... I was seeing some "deadlock" errors in the worker and made a guess it was performance related so I moved the worker and temporal service to a dedicated node group to check that.... I haven't see these errors again but stuff seems to still be hanging. If anyone has any ideas to troubleshoot would be appreciated 🙂

Zack Mattor

10/30/2025, 3:45 PM

Alrighty... I determined the temporal work flows get kicked off and spin....

Copy code

airbyte-temporal-6f5d96f5bb-ttxq8:/etc/temporal$ temporal workflow list
      Status                            WorkflowId                                   Type               StartTime
  Running         check_3c4b322f-c64d-4219-897b-eb17bb1026e0               ConnectorCommandWorkflow   4 minutes ago
  Running         check_aeb5f109-cc8d-467f-973b-785d0cecada0               ConnectorCommandWorkflow   9 minutes ago

Zack Mattor

10/30/2025, 3:46 PM

Seems to get into some kind of loop....

Copy code

airbyte-temporal-6f5d96f5bb-ttxq8:/etc/temporal$ temporal workflow show   --workflow-id check_aeb5f109-cc8d-467f-973b-785d0cecada0
Progress:
  ID           Time                     Type
    1  2025-10-30T15:35:38Z  WorkflowExecutionStarted
    2  2025-10-30T15:35:38Z  WorkflowTaskScheduled
    3  2025-10-30T15:35:38Z  WorkflowTaskStarted
    4  2025-10-30T15:35:38Z  WorkflowTaskCompleted
    5  2025-10-30T15:35:38Z  MarkerRecorded
    6  2025-10-30T15:35:38Z  ActivityTaskScheduled
    7  2025-10-30T15:35:38Z  ActivityTaskStarted
    8  2025-10-30T15:35:38Z  ActivityTaskCompleted
    9  2025-10-30T15:35:38Z  WorkflowTaskScheduled
   10  2025-10-30T15:35:38Z  WorkflowTaskStarted
   11  2025-10-30T15:35:38Z  WorkflowTaskCompleted
   12  2025-10-30T15:35:38Z  ActivityTaskScheduled
   13  2025-10-30T15:35:38Z  ActivityTaskStarted
   14  2025-10-30T15:35:38Z  ActivityTaskCompleted
   15  2025-10-30T15:35:38Z  WorkflowTaskScheduled
   16  2025-10-30T15:35:38Z  WorkflowTaskStarted
   17  2025-10-30T15:35:38Z  WorkflowTaskCompleted
   18  2025-10-30T15:35:38Z  ActivityTaskScheduled
   19  2025-10-30T15:35:38Z  ActivityTaskStarted
   20  2025-10-30T15:35:38Z  ActivityTaskCompleted
   21  2025-10-30T15:35:38Z  WorkflowTaskScheduled
   22  2025-10-30T15:35:38Z  WorkflowTaskStarted
   23  2025-10-30T15:35:38Z  WorkflowTaskCompleted
   24  2025-10-30T15:35:38Z  TimerStarted
   25  2025-10-30T15:36:38Z  TimerFired
   26  2025-10-30T15:36:38Z  WorkflowTaskScheduled
   27  2025-10-30T15:36:38Z  WorkflowTaskStarted
   28  2025-10-30T15:36:38Z  WorkflowTaskCompleted
   29  2025-10-30T15:36:38Z  ActivityTaskScheduled
   30  2025-10-30T15:36:38Z  ActivityTaskStarted
   31  2025-10-30T15:36:38Z  ActivityTaskCompleted
   32  2025-10-30T15:36:38Z  WorkflowTaskScheduled
   33  2025-10-30T15:36:38Z  WorkflowTaskStarted
   34  2025-10-30T15:36:38Z  WorkflowTaskCompleted
   35  2025-10-30T15:36:38Z  TimerStarted
   36  2025-10-30T15:37:38Z  TimerFired
   37  2025-10-30T15:37:38Z  WorkflowTaskScheduled
   38  2025-10-30T15:37:38Z  WorkflowTaskStarted
   39  2025-10-30T15:37:38Z  WorkflowTaskCompleted
   40  2025-10-30T15:37:38Z  ActivityTaskScheduled
   41  2025-10-30T15:37:38Z  ActivityTaskStarted
   42  2025-10-30T15:37:38Z  ActivityTaskCompleted
   43  2025-10-30T15:37:38Z  WorkflowTaskScheduled
   44  2025-10-30T15:37:38Z  WorkflowTaskStarted
   45  2025-10-30T15:37:38Z  WorkflowTaskCompleted
   46  2025-10-30T15:37:38Z  TimerStarted
   47  2025-10-30T15:38:38Z  TimerFired
   48  2025-10-30T15:38:38Z  WorkflowTaskScheduled
   49  2025-10-30T15:38:38Z  WorkflowTaskStarted
   50  2025-10-30T15:38:38Z  WorkflowTaskCompleted
   51  2025-10-30T15:38:38Z  ActivityTaskScheduled
   52  2025-10-30T15:38:38Z  ActivityTaskStarted
   53  2025-10-30T15:38:38Z  ActivityTaskCompleted
   54  2025-10-30T15:38:38Z  WorkflowTaskScheduled
   55  2025-10-30T15:38:38Z  WorkflowTaskStarted
   56  2025-10-30T15:38:38Z  WorkflowTaskCompleted
   57  2025-10-30T15:38:38Z  TimerStarted
   58  2025-10-30T15:39:38Z  TimerFired
   59  2025-10-30T15:39:38Z  WorkflowTaskScheduled
   60  2025-10-30T15:39:38Z  WorkflowTaskStarted
   61  2025-10-30T15:39:38Z  WorkflowTaskCompleted
   62  2025-10-30T15:39:38Z  ActivityTaskScheduled
   63  2025-10-30T15:39:38Z  ActivityTaskStarted
   64  2025-10-30T15:39:38Z  ActivityTaskCompleted
   65  2025-10-30T15:39:38Z  WorkflowTaskScheduled
   66  2025-10-30T15:39:38Z  WorkflowTaskStarted
   67  2025-10-30T15:39:38Z  WorkflowTaskCompleted
   68  2025-10-30T15:39:38Z  TimerStarted
   69  2025-10-30T15:40:38Z  TimerFired
   70  2025-10-30T15:40:38Z  WorkflowTaskScheduled
   71  2025-10-30T15:40:38Z  WorkflowTaskStarted
   72  2025-10-30T15:40:39Z  WorkflowTaskCompleted
   73  2025-10-30T15:40:39Z  ActivityTaskScheduled
   74  2025-10-30T15:40:39Z  ActivityTaskStarted
   75  2025-10-30T15:40:39Z  ActivityTaskCompleted
   76  2025-10-30T15:40:39Z  WorkflowTaskScheduled
   77  2025-10-30T15:40:39Z  WorkflowTaskStarted
   78  2025-10-30T15:40:39Z  WorkflowTaskCompleted
   79  2025-10-30T15:40:39Z  TimerStarted
   80  2025-10-30T15:41:39Z  TimerFired
   81  2025-10-30T15:41:39Z  WorkflowTaskScheduled
   82  2025-10-30T15:41:39Z  WorkflowTaskStarted
   83  2025-10-30T15:41:39Z  WorkflowTaskCompleted
   84  2025-10-30T15:41:39Z  ActivityTaskScheduled
   85  2025-10-30T15:41:39Z  ActivityTaskStarted
   86  2025-10-30T15:41:39Z  ActivityTaskCompleted
   87  2025-10-30T15:41:39Z  WorkflowTaskScheduled
   88  2025-10-30T15:41:39Z  WorkflowTaskStarted
   89  2025-10-30T15:41:39Z  WorkflowTaskCompleted
   90  2025-10-30T15:41:39Z  TimerStarted
   91  2025-10-30T15:42:39Z  TimerFired
   92  2025-10-30T15:42:39Z  WorkflowTaskScheduled
   93  2025-10-30T15:42:39Z  WorkflowTaskStarted
   94  2025-10-30T15:42:39Z  WorkflowTaskCompleted
   95  2025-10-30T15:42:39Z  ActivityTaskScheduled
   96  2025-10-30T15:42:39Z  ActivityTaskStarted

Zack Mattor

10/30/2025, 3:48 PM

I'm thinking it can't spin up the pod for the connection check?.... I see in the temporal data its referencing the proper mysql image for the connection check but I don't see it spin up that container anywhere...

Zack Mattor

10/30/2025, 3:48 PM

@kapa.ai you have any thoughts on the above?

kapa.ai

10/30/2025, 3:48 PM

I'm sorry, but your question "you have any thoughts on the above?" is a bit unclear, as it doesn't specify which issue or topic you're referring to. If you are referencing the most recent knowledge source about the Azure Blob Storage destination connector not deleting files on full refresh overwrite, here’s what I can share: The issue reported is that, after updating to version 1.0.3 of the

destination-azure-blob-storage

connector, files from previous syncs are not being deleted even when using the "Overwrite" stream/object type. The logs show attempts to identify files for deletion using a regex, but there are no errors indicating why the files remain undeleted. A user has tagged a contributor who worked on related files for input, but as of now, there is no official resolution or further insight provided in the knowledge sources. If you are looking for troubleshooting steps or a workaround, unfortunately, the knowledge sources do not contain enough information to provide a solution or detailed thoughts beyond what has already been discussed in the issue thread. If you can clarify your question or specify which topic you are referring to, I’d be happy to help further based on the available information. For now, the best course of action would be to follow the GitHub issue for updates or reach out to the maintainers for more direct support regarding this bug [[destination-azure-blob-storage] Full refresh files are not being deleted](https://github.com/airbytehq/airbyte/issues/61590).

Justin Beasley

10/30/2025, 4:22 PM

did you migrate the temporal database when you migrated the Airbyte DB? (it's two separate databases on the same host) Another thing I'd double-check is the timeout for your ingress (for us, we use a Cloud Load Balancer, so we have to set it on there in addition to the YAML). This can cause issues especially during connection check and schema discovery. I'd recommend setting it high, like 30-60 minutes, to accommodate a lot of possible corner cases (like databases with thousands of tables or really slow APIs and such)

Zack Mattor

10/30/2025, 4:46 PM

I did not migrate the temporal database, I let it re-create that and the temporal_visibility database. I believe that should be fine because I don't need any of the historic workflows that exist in there... but maybe that is causing problems? I do see it creating the temporal workflow just fine, etc via temporal CLI... I'm currently bypassing any form of ingress and forwarding the port directly to the pod...

Justin Beasley

10/30/2025, 4:47 PM

I'm not sure why, but I've always had things be a bit weird when I haven't moved the temporal DBs

Zack Mattor

10/30/2025, 4:47 PM

To sanity check my k8s setup I did just spin up a 100% fresh airbyte install with a new DB and re-created my destination in there manually and it works fine.... I might just migrate my config over to the fresh install...

Justin Beasley

10/30/2025, 5:24 PM

strange. glad at least it's working! maybe something weird with the control plane or secrets or such. we've been on the full Google stack for years (GKE Autopilot, Cloud SQL, Google Secret Manager, Cloud Load Balancer, IAP for Google Account-based auth, etc.) and it's been pretty solid for us, especially with more recent upgrades that are a lot less likely to introduce issues. Airbyte runs Cloud primarily on GCP, so it should be a pretty first-rate citizen. Obviously their deployment has a lot more complexity and customization, but the core should work similarly.

Open in Slack

Previous Next