<@U04S8PLJWMT> What are the possible causes there ...
# ask-ai
w
@kapa.ai What are the possible causes there are if the Workload Launcher is not spinning up any new pods?
k
From the linked issue, there are several observed/likely causes when the Workload Launcher stops spinning up new pods: 1. Workloads stuck in “launched” status - Workloads that have actually completed remain in
launched
in the database. - The claim API refuses to claim workloads that are already in
launched
, so every claim returns
Claimed: false
, and the launcher skips all stages including
CHECK_STATUS
. - This creates a deadlock where workloads never progress and no new pods are started.[[airbyte services stuck](https://github.com/airbytehq/airbyte/issues/68115)] 2. Multiple
dataplane_group
entries after upgrade
- After upgrading to 2.0, some setups ended up with two
dataplane_group
records. - Jobs were pushed to dataplane group A, while the launcher was listening to dataplane group B, so it never saw workloads to launch pods for. - A workaround reported was correcting
dataplane_group_id
in the
workspace
table to point to the active dataplane group.[[airbyte services stuck](https://github.com/airbytehq/airbyte/issues/68115)] 3. Temporal namespace rate limiting / service errors - Temporal logs show
RESOURCE_EXHAUSTED: namespace rate limit exceeded
and
service failures
around
AddWorkflowTask
. - When Temporal is rate-limited or failing to schedule workflow tasks, workflows can get stuck, which in turn prevents new workloads from being launched.[[airbyte services stuck](https://github.com/airbytehq/airbyte/issues/68115)] 4. Workflow thread / cache leaks in Temporal - The issue description mentions “workflow thread can’t be destroyed in time… workflows are getting stuck and can’t be cleaned up, causing memory/cache leaks.” - Stuck workflows and leaked workflow threads can prevent progression of new workflow tasks that would trigger new pods.[[airbyte services stuck](https://github.com/airbytehq/airbyte/issues/68115)] 5. Workload not being claimed at all - Launcher logs repeatedly show
Claimed: false for workload ...
and
Workload not claimed. Setting SKIP flag to true. SKIP Stage: LOAD_SHED / MUTEX / CHECK_STATUS
. - If every workload returns
Claimed: false
, no subsequent pipeline stages (including pod creation and status checks) run, so no new pods appear.[[airbyte services stuck](https://github.com/airbytehq/airbyte/issues/68115)] Beyond these points, the knowledge source does not provide a fully confirmed root cause or official fix, only observed behavior and one SQL-level workaround for the dataplane group mismatch.