consistent query buffer is full in self-hosted airbyte core

New message text here

@kapa.ai Why does tctl taskqueue describe show no pollers for CONNECTION_UPDATER even though the worker pod logs show it registered the poller successfully? Airbyte v2.0.1, K3s, Helm chart 2.0.19

The knowledge sources provided don't contain specific information about

tctl taskqueue describe

showing no pollers for

CONNECTION_UPDATER

despite successful poller registration in worker logs for Airbyte v2.0.1 on K3s with Helm chart 2.0.19. However, there is a related pattern documented in the knowledge sources that may be relevant to your situation: Possible Related Issue: Dataplane Group Mismatch One documented issue with Airbyte 2.0.x (including 2.0.1) involves the workload-launcher registering under a different dataplane group than the one workloads are routed to, causing syncs to be stuck in "Queued" with "workload was never claimed." [Dataplane Group Mismatch] The symptoms overlap with what you're describing — services appear to register successfully in logs, but work is never actually picked up. The workaround documented is:

-- Move all dataplanes to the AUTO group
UPDATE dataplane
SET dataplane_group_id = (
  SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1
)
WHERE dataplane_group_id != (
  SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1
);

Then restart the workload-launcher:

kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n <your-namespace>

[Dataplane Group Mismatch] Also Worth Checking There are also known issues with

NonDeterministicException

errors from stale Temporal

ConnectionManagerWorkflow

instances carried over from older versions, which can cause workers to appear registered but not function correctly. [Dataplane Group Mismatch] I'm sorry, but the knowledge sources don't contain enough specific information to directly explain the

tctl taskqueue describe

discrepancy you're seeing. I'd recommend checking the Airbyte Community Slack for more targeted help on this specific scenario. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai After helm uninstall + install with fresh temporal DB (dropped temporal and temporal_visibility), Temporal is stopping all physicalTaskQueueManagers with 'Stopped cause: Idle', including CONNECTION_UPDATER. Worker registers pollers but Temporal immediately shuts them down. Sync now gives DEADLINE_EXCEEDED. Airbyte v2.0.1, chart 2.0.19, K3s. Is there a Temporal config to prevent idle task queue shutdown?

The knowledge sources don't contain specific information about preventing idle

physicalTaskQueueManager

shutdowns via Temporal configuration in your exact scenario (fresh DB after helm reinstall on K3s with Airbyte v2.0.1/chart 2.0.19). However, there is directly relevant evidence from the knowledge sources about Temporal dynamic configuration overrides that may help: Temporal Dynamic Config Override One documented approach for addressing Temporal instability in self-hosted Airbyte (Kubernetes/Helm) is to override Temporal's dynamic configuration. Here's the pattern used: [sync attempt hangs issue] Step 1: Create a ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: airbyte-platform-temporal-dynamicconfig-override
  namespace: airbyte-prod
data:
  "development.yaml": |
    frontend.namespaceCount:
      - value: 4096
        constraints: {}
    frontend.namespaceRPS:
      - value: 76800
        constraints: {}
    frontend.persistenceMaxQPS:
      - value: 5000
        constraints: {}
    history.historyMgrNumConns:
      - value: 50
        constraints: {}
    system.advancedVisibilityWritingMode:
      - value: "off"
        constraints: {}
    limit.blobSize.error:
      - value: 15728640 # 15MB
        constraints: {}
    limit.blobSize.warn:
      - value: 10485760 # 10MB
        constraints: {}

Step 2: Mount it in the Temporal pod via Helm values

temporal:
  extraEnv:
    - name: DYNAMIC_CONFIG_FILE_PATH
      value: config/dynamicconfig-override/development.yaml
  extraVolumeMounts:
    - name: airbyte-temporal-dynamicconfig-override
      mountPath: "/etc/temporal/config/dynamicconfig-override/"
  extraVolumes:
    - name: airbyte-temporal-dynamicconfig-override
      configMap:
        name: airbyte-platform-temporal-dynamicconfig-override
        items:
          - key: development.yaml
            path: development.yaml

[sync attempt hangs issue] About the Stopped cause: Idle Behavior The knowledge sources show that

physicalTaskQueueManager

stopping with

cause: Idle

is a normal Temporal behavior — it stops managing a task queue when there are no active pollers consuming from it. [Temporal context deadline exceeded] The issue is likely that your worker pollers are registering but then disconnecting or failing before Temporal can route work to them, rather than Temporal aggressively shutting down queues. Also Worth Checking One documented case found that Temporal is highly sensitive to database latency — even when the DB appears healthy, network distance between the Temporal pod and the PostgreSQL instance caused

context deadline exceeded

errors. The fix was placing the RDS instance in the same region/zone as the cluster. [Temporal context deadline exceeded] The knowledge sources don't contain a specific Temporal config parameter to tune the idle task queue shutdown timeout for your exact scenario. I'd recommend checking the Airbyte Community Slack for more targeted guidance. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Airbyte v2.0.1 self-hosted on K3s — no sync works at all. Every sync attempt gives HTTP 504. Fresh install, fresh temporal DB. Worker starts fine, no errors. What's the recommended troubleshooting path?

Based on the knowledge sources and your conversation history, here is the recommended troubleshooting path for Airbyte v2.0.1 on K3s with all syncs failing with HTTP 504. 1. Check for Dataplane Group Mismatch (Most Likely Cause) This is the most commonly documented root cause for fresh installs of Airbyte 2.0.x where syncs never start. The bootloader can create two dataplane groups, and the workload-launcher registers under one while workloads are routed to the other. [Dataplane Group Mismatch] Connect to your Airbyte PostgreSQL database and run:

SELECT id, name FROM dataplane_group;

If you see two rows (one named

AUTO

and one unnamed/empty), you have the mismatch. Fix it with:

UPDATE dataplane
SET dataplane_group_id = (
  SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1
)
WHERE dataplane_group_id != (
  SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1
);

Then restart the workload-launcher:

kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n <your-namespace>

After the fix, healthy launcher logs should show:

INFO  Resuming ApiQueuePoller with <AUTO-group-id> and high
INFO  Resuming ApiQueuePoller with <AUTO-group-id> and default

[Dataplane Group Mismatch] 2. Check Workload-Launcher Logs

kubectl logs -l app.kubernetes.io/name=workload-launcher --namespace <airbyte-namespace> --tail=100

If you only see

PodSweeper

cycle logs and no

ApiQueuePoller

initialization, the dataplane mismatch is confirmed. [Dataplane Group Mismatch] 3. Check Workload Status in the Database

SELECT id, status, termination_reason FROM workload WHERE status = 'pending' LIMIT 10;

If

termination_reason

shows

"The workload was never claimed"

, this confirms the launcher is not picking up work. [Dataplane Group Mismatch] 4. Alternative Fix (Without Direct DB Access) If you can't access the database directly, you can use the internal API from within the cluster:

# Get the correct group from the dataplanes API
CORRECT_GROUP=$(kubectl exec -n <NS> deploy/airbyte-server -- \
  curl -s http://localhost:8001/api/public/v1/dataplanes \
  | python3 -c "import sys,json; print(json.load(sys.stdin)[0]['regionId'])")

# Update the workspace to match
kubectl exec -n <NS> deploy/airbyte-server -- \
  curl -s -X POST http://localhost:8001/api/v1/workspaces/update \
  -H "Content-Type: application/json" \
  -d "{\"workspaceId\":\"<WORKSPACE_ID>\",\"dataplaneGroupId\":\"$CORRECT_GROUP\"}"

[Dataplane Group Mismatch] 5. Prevent Recurrence on Future Installs Add this to your Helm values before upgrading or reinstalling:

global:
  dataplaneGroups:
    defaultDataplaneGroupName: "AUTO"

This prevents the bootloader from creating a mismatched unnamed group. [airbyte services stuck] Given your fresh install scenario and the 504 errors with no sync activity, the dataplane group mismatch is the most likely culprit based on documented cases. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Airbyte v2.0.1 on K3s single node — ConnectionManagerWorkflow is created in Temporal (WorkflowTaskScheduled) but never starts (no WorkflowTaskStarted event). Worker logs show successful poller registration, no errors. Launcher logs are healthy with correct AUTO dataplane group. Server gets DEADLINE_EXCEEDED. Fresh helm install with fresh temporal DB. This happens with every connection. What causes the worker to register pollers but never pick up workflow tasks?

Based on the knowledge sources available, I can point to one directly documented case that matches your symptom pattern closely. Temporal Database Latency / Network Distance The most relevant documented case for your exact symptom — pollers register successfully,

WorkflowTaskScheduled

is created but

WorkflowTaskStarted

never fires, and

DEADLINE_EXCEEDED

results — is Temporal being highly sensitive to latency between the Temporal pod and its PostgreSQL database. [Temporal context deadline exceeded] In that case: • The database appeared healthy (migrations ran, rows inserted without issue) • The only DB-side error was a benign duplicate key violation • Temporal pods logged

context deadline exceeded

continuously • The fix was placing the PostgreSQL instance in the same region/zone as the cluster (in that case, same AWS region as EKS) The documented conclusion was: > "It seems that the Temporal pod is highly sensitive to delays and timeouts when interacting with the database. However, I couldn't find any environment configuration that could mitigate this issue." [Temporal context deadline exceeded] What to Check 1. Database network latency — Even if your fresh Temporal DB appears healthy, measure the round-trip latency between the Temporal pod and PostgreSQL. On a K3s single-node setup this should be negligible if both are local, but if your DB is external (e.g., RDS or a remote host), network distance could be the cause. 2. Temporal persistence errors in Temporal pod logs — Look specifically for:"error":"context deadline exceeded" "error-type":"persistence.AppendHistoryTimeoutError"These indicate Temporal cannot write workflow state fast enough. [sync pulling 0 records] 3. Temporal pod resource constraints — Check if the Temporal pod itself is CPU/memory throttled, which can cause it to miss the window to start workflow tasks after scheduling them. [sync pulling 0 records] Important Limitation The knowledge sources do not contain a documented case that exactly matches your combination of: fresh DB, healthy launcher with correct AUTO group, successful poller registration, and

WorkflowTaskScheduled

with no

WorkflowTaskStarted

. The sources don't provide enough information to give a definitive root cause beyond the database latency pattern above. I'd recommend joining the Airbyte Community Slack for more targeted help on this specific scenario. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Temporal logs show 'consistent query buffer is full' continuously. Only 2 connections are enabled (rest are paused). Server appears to poll ConnectionManagerWorkflow state in a loop, filling the buffer before the worker can process any workflow tasks. Is this expected behavior with just 2 enabled connections? Is there a way to reduce the polling frequency or increase the query buffer size?

The "consistent query buffer is full" error with only 2 enabled connections is not expected behavior — this points to something generating an abnormally high query rate against Temporal, not the connections themselves. What the Knowledge Sources Say The error means Temporal's query buffer is saturated because queries are arriving faster than workflows can process them: >

"consistent query buffer is full, this may be caused by too many queries and workflow not able to process query fast enough"

[timeout errors issue] Most Likely Cause in Your Scenario Based on your conversation history, the

WorkflowTaskScheduled

events are never being followed by

WorkflowTaskStarted

. This means the

ConnectionManagerWorkflow

is stuck — it cannot make progress. When the Airbyte server polls the workflow state (via Temporal's consistent query mechanism) and the workflow is stuck, each poll adds to the buffer without being drained. With a tight polling loop, this fills the buffer rapidly even with just 2 connections. [JobStatus FAILED issue] A documented precedent: a single repeated error logged millions of times eventually caused the same "consistent query buffer is full" state, making Airbyte unresponsive — the root cause was a stuck workflow, not the number of connections. [JobStatus FAILED issue] What You Can Try 1. Increase the Temporal query buffer via dynamic config override The knowledge sources document a Temporal dynamic config override pattern for self-hosted Airbyte on Kubernetes/Helm. You can apply it to tune Temporal's behavior: [sync attempt hangs issue]

apiVersion: v1
kind: ConfigMap
metadata:
  name: airbyte-platform-temporal-dynamicconfig-override
  namespace: <your-namespace>
data:
  "development.yaml": |
    frontend.namespaceRPS:
      - value: 76800
        constraints: {}
    frontend.persistenceMaxQPS:
      - value: 5000
        constraints: {}
    frontend.throttledLogRPS:
      - value: 200
        constraints: {}

Mount it in the Temporal pod:

temporal:
  extraEnv:
    - name: DYNAMIC_CONFIG_FILE_PATH
      value: config/dynamicconfig-override/development.yaml
  extraVolumeMounts:
    - name: airbyte-temporal-dynamicconfig-override
      mountPath: "/etc/temporal/config/dynamicconfig-override/"
  extraVolumes:
    - name: airbyte-temporal-dynamicconfig-override
      configMap:
        name: airbyte-platform-temporal-dynamicconfig-override
        items:
          - key: development.yaml
            path: development.yaml

[sync attempt hangs issue] 2. Address the root cause first The buffer filling is a symptom of the underlying issue you've been troubleshooting —

WorkflowTaskScheduled

with no

WorkflowTaskStarted

. Until the worker can actually pick up and execute workflow tasks, the query buffer will keep filling regardless of its size. The dynamic config override may buy some headroom, but won't fix the core problem. The knowledge sources do not contain specific parameters to reduce the Airbyte server's polling frequency against Temporal's consistent query API. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Are there known regressions in Airbyte 2.0.1 (chart 2.0.19) vs 2.0.0 (chart 2.0.18) related to Temporal workflow processing? WorkflowTaskStarted never fires after WorkflowTaskScheduled on CONNECTION_UPDATER queue. This is a fresh install with fresh temporal DB. Worker registers pollers successfully. Considering downgrade to 2.0.0

The knowledge sources do not contain any documented information about specific regressions between Airbyte 2.0.0 (chart 2.0.18) and 2.0.1 (chart 2.0.19) related to Temporal workflow processing or the

WorkflowTaskStarted

never firing after

WorkflowTaskScheduled

on the

CONNECTION_UPDATER

queue. What the knowledge sources do document about chart 2.0.18 vs 2.0.0 is limited to deployment/compatibility guidance: • Helm chart version 2.0.18 is the correct chart for Airbyte platform version 2.0.0 (the chart and app version numbers differ). [Multiple issues with 2.0.0 chart] • Users confirmed that chart 2.0.18 works correctly for Airbyte 2.0, while the older

airbyte/airbyte

chart repo version

2.0.0

does not. [Multiple issues with 2.0.0 chart] There is no documented evidence in the knowledge sources of a regression specifically in 2.0.1/chart 2.0.19 that would cause

WorkflowTaskStarted

to never fire on a fresh install with a fresh Temporal DB. Given your conversation history, the symptoms you're describing (pollers register,

WorkflowTaskScheduled

created but

WorkflowTaskStarted

never fires,

DEADLINE_EXCEEDED

) are consistent with the Temporal database latency pattern documented for Airbyte 2.0.x on Kubernetes — not a version-specific regression. [Temporal context deadline exceeded] I'd recommend checking the Airbyte Community Slack or the GitHub issues for any reported regressions specific to chart 2.0.19 before deciding to downgrade, as the knowledge sources don't provide enough evidence to confirm or rule out a version-specific regression. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.