https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I have deployed airbyte 2.0 on kube...
# replication-ask-ai
f
@kapa.ai I have deployed airbyte 2.0 on kubernetes using as db an external DB that was previously used with airbyte version 1.5. On airbyte UI now I see all sources/destinations/connections from the old 1.5 server but on worker I have the following errors and I cannot proceed neither creating new sources. How can I resolve?
Copy code
2025-12-15 00:00:43,067 [Workflow Poller taskQueue="SYNC", namespace="default": 5]	WARN	i.t.i.w.Poller$PollerUncaughtExceptionHandler(logPollErrors):362 - Failure in poller thread Workflow Poller taskQueue="SYNC", namespace="default": 5
io.grpc.StatusRuntimeException: CANCELLED: context canceled
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:351)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:332)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174)
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollWorkflowTaskQueue(WorkflowServiceGrpc.java:5634)
	at io.temporal.internal.worker.WorkflowPollTask.doPoll(WorkflowPollTask.java:174)
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:155)
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:49)
	at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:336)
	at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:296)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2025-12-15 00:02:41,868 [Workflow Poller taskQueue="ui_commands", namespace="default": 2]	WARN	i.t.i.w.Poller$PollerUncaughtExceptionHandler(logPollErrors):362 - Failure in poller thread Workflow Poller taskQueue="ui_commands", namespace="default": 2
io.grpc.StatusRuntimeException: CANCELLED: context canceled
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:351)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:332)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174)
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollWorkflowTaskQueue(WorkflowServiceGrpc.java:5634)
	at io.temporal.internal.worker.WorkflowPollTask.doPoll(WorkflowPollTask.java:174)
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:155)
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:49)
	at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:336)
	at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:296)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2025-12-14 19:36:30,863 [Workflow Poller taskQueue="CONNECTION_UPDATER", namespace="default": 4]	WARN	i.t.i.w.Poller$PollerUncaughtExceptionHandler(logPollErrors):362 - Failure in poller thread Workflow Poller taskQueue="CONNECTION_UPDATER", namespace="default": 4
io.grpc.StatusRuntimeException: CANCELLED: context canceled
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:351)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:332)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:174)
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.pollWorkflowTaskQueue(WorkflowServiceGrpc.java:5634)
	at io.temporal.internal.worker.WorkflowPollTask.doPoll(WorkflowPollTask.java:174)
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:155)
	at io.temporal.internal.worker.WorkflowPollTask.poll(WorkflowPollTask.java:49)
	at io.temporal.internal.worker.Poller$PollExecutionTask.run(Poller.java:336)
	at io.temporal.internal.worker.Poller$PollLoopTask.run(Poller.java:296)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
k
Airbyte 2.x is not compatible at the database schema level with Airbyte 1.5.x, and the behavior you’re seeing (old objects visible in the UI but workers/Temporal failing) is consistent with using a DB that was initialized by an older version. From the provided issues and forum threads, similar Temporal errors (
DEADLINE_EXCEEDED
, 
context deadline exceeded
, poller failures) appear when: - Temporal cannot reliably access its database or metadata. [[External DB deploy](https://discuss.airbyte.io/t/2179); [Temporal context deadline](https://github.com/airbytehq/airbyte/issues/59730)] - The deployment was effectively “half‑migrated”, or resources/services restart in a bad state and Temporal pollers end up cancelled/unavailable. [[Temporal context deadline](https://github.com/airbytehq/airbyte/issues/59730); [Airbyte services stuck](https://github.com/airbytehq/airbyte/issues/68115)] However, the knowledge you provided does not contain a documented, supported path to reuse an Airbyte 1.5 Postgres schema directly with Airbyte 2.0, nor any migration guide that would make this safe. Because of that, I can’t point to an official way to fix your current DB in‑place. Given what’s documented, the safest way forward (and what I can recommend based on the sources) is: 1. Do not reuse the 1.5 DB directly with 2.0. Use a fresh database/schema for the 2.0 deployment and let Airbyte 2.0 initialize its own schema. The external‑DB Temporal timeout examples you shared were solved via redeploys and “clean” environments rather than trying to repair a mismatched schema. [[Temporal context deadline](https://github.com/airbytehq/airbyte/issues/59730); [External DB deploy](https://discuss.airbyte.io/t/2179)] 2. Standing up a clean 2.0 environment: - Create a new Postgres DB (or new schema) for Airbyte 2.0. - Point your Helm/Kubernetes deployment at this new DB and redeploy so server, worker, and temporal all start from a consistent, fresh state. Similar redeploys resolved Temporal poller issues in the EKS example. [[Temporal context deadline](https://github.com/airbytehq/airbyte/issues/59730)] 3. Recreate objects from 1.5: The knowledge sources don’t describe an automated migration of sources/destinations/connections from 1.5 → 2.0. So, based on what’s available, the only clearly safe approach is: - Export connection/source/destination configs from the 1.5 environment (if you still have it) or manually recreate them in the fresh 2.0 environment. - Then test new source creation and syncs in 2.0. 4. If you must keep the existing DB: The knowledge sources don’t provide a supported procedure to repair the DB for 2.0, or to interpret these Temporal 
CANCELLED: context canceled
errors specifically as a known upgrade issue, so I can’t give a reliable step‑by‑step fix here. I’d strongly recommend opening a GitHub issue and explicitly stating that a 1.5 DB was reused for 2.0, so the team can confirm whether any internal migration tooling exists (none is mentioned in the documents you shared). In summary, based on the available information, the recommended resolution is to: - Deploy Airbyte 2.0 against a new, empty Postgres DB, - Let it initialize cleanly, - Then recreate (or import, if you have an export) your sources/destinations/connections rather than pointing 2.0 at the 1.5 DB.
f
@kapa.ai how can I clean postgres internal DB? Do I have to reinstall airbyte when cleaned?
k
A message from kapa.ai
You have two different “clean up” options, with different consequences. --- ### 1. Wipe the internal Postgres DB completely (start from scratch) This is equivalent to reinstalling Airbyte from a data perspective: 1. Back up first (recommended in case you regret it). [[Helm V2 upgrade guide](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#how-to-upgrade)] 2. Drop and recreate the schema inside the Airbyte DB:
Copy code
bash    # copy your backup in if you made one    kubectl cp ./airbyte_backup.sql airbyte/airbyte-db-0:/tmp/airbyte_backup.sql       # drop and recreate the public schema (erases all data)    kubectl exec -n airbyte -it airbyte-db-0 -- bash -c \      "psql -U airbyte -d db-airbyte -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'"
[[Drop/recreate schema](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] 3. Restart the Airbyte pods (or redeploy via Helm) so the platform can re‑initialize the DB on next start. - This is effectively like a fresh install: you will lose all sources, destinations, connections, states, etc. - You don’t strictly have to reinstall the chart if you just drop the schema, but many users pair this with a redeploy/helm uninstall+install to be sure everything starts clean. [[Helm reinstall steps](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] --- ### 2. Clear only synced data in destinations (keep config & metadata) If your goal is to remove synced data but keep your connections/sources/destinations, you should use Clear (formerly “Reset”) at the connection/stream level instead of touching Postgres: - In the UI: - Go to a connection → Settings → Clear data to wipe all streams, or - On the connection status page, use the three‑dot menu next to a stream → Clear data to wipe just that stream. [[Clearing data](https://docs.airbyte.com/platform/operator-guides/clear)] - After a Clear completes, run a sync to backfill. [[Clearing data](https://docs.airbyte.com/platform/operator-guides/clear)] This does not clean the internal DB; it cleans destination tables/files. --- ### Which should you use? - Use Clear if you only want to delete synced data in destinations. - Use schema drop/recreate (option 1) if you truly want to reset the whole Airbyte instance to an empty state. After doing so, expect to recreate everything (or restore from backup).
f
@kapa.ai I have these pods. can you please list the steps to 1. erase data oon postgres 2. restart airbyte recreating database objects
Copy code
(base) ➜  ~ kubectl get pods -n airbyte
NAME                                                  READY   STATUS    RESTARTS      AGE
airbyte-minio-0                                       1/1     Running   0             3h42m
airbyte-v2-connector-builder-server-666d68454-d5j7d   1/1     Running   0             39h
airbyte-v2-cron-c646874cc-prdwd                       1/1     Running   0             39h
airbyte-v2-server-7b4d756955-9zmpg                    1/1     Running   0             39h
airbyte-v2-temporal-7f6b6698d6-8bmsx                  1/1     Running   0             39h
airbyte-v2-worker-6f4f78f464-b7lxj                    1/1     Running   0             39h
airbyte-v2-workload-api-server-6c94876b5-lp654        1/1     Running   0             39h
airbyte-v2-workload-launcher-5577db5447-w4sp9         1/1     Running   1 (39h ago)   39h
k
A message from kapa.ai
You’re currently using **MinIO only**; there is no Postgres pod in the 
airbyte
namespace, which means: - Either you configured an external Postgres DB (outside the cluster), or - You installed Airbyte without the internal 
airbyte-db-0
pod for some other reason. The “drop schema and recreate” operations in the docs are run inside the Postgres pod (e.g. 
airbyte-db-0
). [[Backup/restore DB](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] Because we don’t see a Postgres pod, I cannot safely give you exact 
kubectl exec
commands for your setup from the knowledge sources alone. The steps below show the pattern from the docs; you’ll need to adapt them to your actual Postgres deployment (pod name, DB name, DB user), which isn’t visible in the snippet you shared. --- ## 1. Erase data on Postgres From the docs, when using the internal DB, they: 1. Copy a backup file into the pod (optional). 2. Drop and recreate the 
public
schema to erase all data. 3. Optionally restore from backup. [[Backup/restore DB](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] Example for internal DB (you must adjust pod name / namespace / DB name / user for your own Postgres):
Copy code
bash # 2) Drop and recreate the schema (this erases all data) kubectl exec -n airbyte -it airbyte-db-0 -- bash -c \   "psql -U airbyte -d db-airbyte -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'"
[[Drop/recreate schema](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] In your case (external DB): - Identify how Postgres is deployed (e.g. managed RDS, another namespace, or another cluster). - Run the same SQL (
DROP SCHEMA public CASCADE; CREATE SCHEMA public;
) against the Airbyte database using 
psql
or your DB admin tool. - Make sure you are acting on the correct DB/schema that Airbyte is using (check your Helm 
values.yaml
global.database.*
or your external DB connection settings). [[External DB config](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database#values)] > Warning: this fully wipes all Airbyte data (sources, destinations, connections, jobs, state, etc.) from that DB. --- ## 2. Restart Airbyte so it recreates DB objects From the Helm V2 upgrade docs, when you keep using the same namespace and database, the flow is: 1. Uninstall the Helm release (optionally; namespace remains). 2. Reinstall the chart, pointing at the same DB; Airbyte initializes schema. [[Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] If you’re using Helm V2 community/Core charts, the pattern is:
Copy code
bash # (Optional but clean) uninstall the existing release, *keep* namespace helm uninstall airbyte -n airbyte  # Reinstall, using your existing values.yaml and same external DB settings helm install airbyte airbyte-v2/airbyte \   --namespace airbyte \   --values ./values.yaml \   --version <your-chart-version>
[[Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] If you don’t want to uninstall/reinstall: - After you drop/recreate the schema in Postgres, you can simply restart the pods so server/worker/Temporal re‑run migrations on an empty DB:
Copy code
bash kubectl rollout restart deployment -n airbyte airbyte-v2-server kubectl rollout restart deployment -n airbyte airbyte-v2-worker kubectl rollout restart deployment -n airbyte airbyte-v2-temporal kubectl rollout restart deployment -n airbyte airbyte-v2-cron kubectl rollout restart deployment -n airbyte airbyte-v2-connector-builder-server kubectl rollout restart deployment -n airbyte airbyte-v2-workload-api-server kubectl rollout restart deployment -n airbyte airbyte-v2-workload-launcher
This exact 
rollout restart
sequence is not explicitly documented in the sources, but it follows directly from the documented principle that the DB is re‑initialized on startup after schema drop. [[Drop/recreate schema](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] --- ### Summary Because your Postgres is external, the docs don’t show your exact pod/host names. The documented pattern is: 1. On the Airbyte DB: run 
DROP SCHEMA public CASCADE; CREATE SCHEMA public;
to erase all data. [[Drop/recreate schema](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] 2. Restart or reinstall the Airbyte Helm chart so that server/worker/Temporal connect to this now‑empty DB and recreate all Airbyte database objects. [[Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] If you can share how your Postgres is deployed (Helm release name / namespace, or RDS/CloudSQL, etc.), I can help you map these doc patterns to concrete commands.
f
@kapa.ai having this situation after restart
Copy code
(base) ➜  ~ kubectl get pods -n airbyte
NAME                                                   READY   STATUS                   RESTARTS      AGE
airbyte-minio-0                                        1/1     Running                  0             3h52m
airbyte-v2-connector-builder-server-7c4f9f4987-hthsx   1/1     Running                  0             4m39s
airbyte-v2-cron-7888c9cdfc-kt5cc                       0/1     OutOfmemory              0             4m40s
airbyte-v2-cron-7888c9cdfc-skvst                       1/1     Running                  0             4m
airbyte-v2-server-76f8798cc7-64hdq                     0/1     Running                  4 (19s ago)   4m21s
airbyte-v2-temporal-7dcbb766b7-n7ghs                   0/1     ContainerStatusUnknown   0             4m40s
airbyte-v2-temporal-7dcbb766b7-rjsdw                   1/1     Running                  0             4m35s
airbyte-v2-worker-58cfb97cc-2cjsh                      1/1     Running                  0             4m
airbyte-v2-worker-58cfb97cc-hgt8b                      0/1     OutOfmemory              0             4m41s
airbyte-v2-workload-api-server-d8b76d46d-mk6fg         0/1     Error                    0             4m38s
airbyte-v2-workload-api-server-d8b76d46d-mqdw8         1/1     Running                  0             4m1s
airbyte-v2-workload-launcher-5fc484c76f-t2v9n          0/1     CrashLoopBackOff         4 (65s ago)   4m36s
k
A message from kapa.ai
The pod statuses point to two separate issues: - Resource problems (
OutOfmemory
, 
CrashLoopBackOff
, 
Error
) - A likely Temporal / DB initialization issue after you wiped the database The knowledge sources you provided don’t include detailed troubleshooting steps for OutOfMemory or CrashLoopBackOff beyond the general guidance to redeploy and ensure a clean DB and environment. They also don’t show the expected logs for these specific pods in Airbyte 2.0, so I can’t give an exact root cause from documentation alone. Given that, here is what is clearly supported by the docs and related issues, and what you should check next: --- ### 1. Confirm DB and Temporal are healthy After dropping the schema, Airbyte needs to recreate all database objects on startup. If Temporal or the DB are misconfigured or too slow to initialize, you can see persistent errors and poller issues. Similar symptoms are mentioned in external‑DB deployments where Temporal times out or pods stay unhealthy until the deployment is fixed. [[External DB config](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database); [External DB deploy issue](https://discuss.airbyte.io/t/3787)] Please verify: 1. External Postgres settings in your `values.yaml`: - 
postgresql.enabled: false
- 
global.database.type: external
- 
global.database.host
, 
port
, and 
name
(or 
database
for older chart) match your DB - 
global.database.secretName
points to a secret containing 
database-user
and 
database-password
keys (for chart V2). [[External DB config](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database#values)] 2. That you can connect from a pod to the DB (e.g., by running 
psql
in a debug container). The docs don’t show this command, but for any external DB configuration this is a critical sanity check. 3. 
airbyte-v2-temporal
has at least one Running replica (you do have one Running pod, which is good). If configuration is wrong, fix 
values.yaml
and redeploy via Helm, as described in the Helm V2 docs. [[Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] --- ### 2. Look into logs of failing pods Because the docs don’t describe these exact failure states, you’ll need to inspect logs to know if this is: - A DB/Temporal initialization issue, or - Purely resource limits (e.g., Java out of memory), or - Something else. Run (examples):
Copy code
bash kubectl logs -n airbyte airbyte-v2-server-76f8798cc7-64hdq kubectl logs -n airbyte airbyte-v2-temporal-7dcbb766b7-n7ghs kubectl logs -n airbyte airbyte-v2-workload-api-server-d8b76d46d-mk6fg kubectl logs -n airbyte airbyte-v2-workload-launcher-5fc484c76f-t2v9n kubectl logs -n airbyte airbyte-v2-cron-7888c9cdfc-kt5cc kubectl logs -n airbyte airbyte-v2-worker-58cfb97cc-hgt8b
Check specifically for: - Connection failures to Postgres - Temporal errors mentioning connection, namespace, or persistence problems (similar to the poller errors you saw earlier) - Java OOM messages Unfortunately, the knowledge sources don’t document specific log patterns for these pods in 2.0, so I can’t tell you “if you see X, do Y” from docs alone. --- ### 3. Address OutOfMemory / CrashLoopBackOff The docs you shared don’t include resource‑tuning guidance, but 
OutOfmemory
indicates that: - The container’s memory limit is too low for the current workload, or - The node is under pressure and the pod is evicted. Actions (based on Kubernetes best practices, not specifically documented for Airbyte in your sources): - Increase memory requests/limits for the affected components in your 
values.yaml
, then 
helm upgrade
/ 
helm install
again. [[Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] - Ensure your nodes have enough free memory. Because there is no explicit Airbyte 2.0 resource recommendation in the provided docs, I can’t state exact numbers. --- ### 4. When to reinstall vs. just restart The upgrade / reinstall guides show that it is safe to: - 
helm uninstall
the release while keeping the namespace, then - 
helm install
again with your 
values.yaml
, pointing to the (now empty) DB. [[How to upgrade](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#how-to-upgrade); [Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] This forces all deployments to be recreated and lets Airbyte initialize the empty DB from scratch. If after fixing DB config and resource limits you still see crashes, a clean uninstall/reinstall (keeping the namespace and DB) is a reasonable next step per the docs:
Copy code
bash helm uninstall airbyte -n airbyte  helm install airbyte airbyte-v2/airbyte \   --namespace airbyte \   --values ./values.yaml \   --version <your-chart-version>
[[How to upgrade](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#how-to-upgrade); [Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] --- ### What I can’t answer from the docs The knowledge sources do not: - Describe Airbyte 2.0 pod resource defaults or tuning steps for OOM. - Provide a troubleshooting matrix for 
CrashLoopBackOff
on 
workload-launcher
or 
workload-api-server
. - Offer a documented, supported “wipe DB and auto‑migrate” process after jumping from 1.5 to 2.0 on the same DB. So I can’t confidently give a precise, documented single fix. The next practical steps, consistent with the docs, are: 1. Double‑check external DB configuration in 
values.yaml
and DB connectivity. [[External DB config](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database)] 2. Inspect logs of the failing pods for DB/Temporal errors. 3. Increase memory limits for the OOM pods and redeploy via Helm. [[Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)] 4. If problems persist even with correct DB config and adequate resources, perform a clean Helm uninstall/install in the same namespace, pointing to the empty DB, as shown in the Helm V2 upgrade docs. [[How to upgrade](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#how-to-upgrade)] If you can paste the logs from one or two of the failing pods (
server
, 
temporal
, 
workload-launcher
), I can help interpret them against the general patterns from the Temporal/DB issues you shared.
f
@kapa.ai having this error on one of temporal pods
Copy code
{
  "level": "error",
  "ts": "2025-12-15T08:58:19.891Z",
  "msg": "Operation failed with internal error.",
  "error": "GetTasks operation failed. Failed to get rows. Error: sql: database is closed",
  "error-type": "serviceerror.Unavailable",
  "operation": "GetTasks",
  "logging-call-at": "/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:1319",
  "stacktrace": "<http://go.temporal.io/server/common/log.(*zapLogger).Error|go.temporal.io/server/common/log.(*zapLogger).Error>\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:154\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:1319\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:1291\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).GetTasks.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:557\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).GetTasks\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_metric_clients.go:559\ngo.temporal.io/server/common/persistence.(*taskRetryablePersistenceClient).GetTasks.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_retryable_clients.go:654\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/backoff/retry.go:89\ngo.temporal.io/server/common/persistence.(*taskRetryablePersistenceClient).GetTasks\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/persistence_retryable_clients.go:658\ngo.temporal.io/server/service/matching.(*taskQueueDB).GetTasks\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/matching/db.go:289\ngo.temporal.io/server/service/matching.(*taskReader).getTaskBatchWithRange\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/matching/task_reader.go:249\ngo.temporal.io/server/service/matching.(*taskReader).getTaskBatch\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/matching/task_reader.go:276\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/home/runner/work/docker-builds/docker-builds/temporal/service/matching/task_reader.go:202\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/home/runner/work/docker-builds/docker-builds/temporal/internal/goro/group.go:58"
}
k
A message from kapa.ai
This Temporal error means the Temporal server lost its connection to the Postgres persistence store: > 
GetTasks operation failed. Failed to get rows. Error: sql: database is closed
In the docs and issues you shared, similar Temporal errors (
context deadline exceeded
, timeouts, persistence failures) are always rooted in DB connectivity / configuration / network problems, especially when using an external Postgres. [[External DB deploy](https://discuss.airbyte.io/t/2179); [Temporal context deadline](https://github.com/airbytehq/airbyte/issues/59730)] Given that you dropped the schema and then restarted, the most likely causes (based on those references) are: 1. Temporal can’t reliably reach the Postgres instance - Wrong host/port, credentials, or SSL settings for the external DB. - Network path issues (e.g., VPC/firewall/security group, region mismatch). Similar problems produced persistence errors and task queue failures with external RDS/Azure Postgres. [[External DB deploy](https://discuss.airbyte.io/t/2179); [Azure Postgres SSL](https://discuss.airbyte.io/t/2928)] 2. The DB or connection was restarted/closed under Temporal When the Temporal process tries to read tasks and finds “database is closed”, it usually means the underlying DB connection pool has been torn down due to an error (connection refused, timeout, SSL, etc.), not that the schema is wrong. The knowledge sources don’t show this exact “sql: database is closed” message, but they do show Temporal failing similarly when: - The DB user lacks permissions or the DB is missing. [[Unable to create SQL DB](https://discuss.airbyte.io/t/1544); [Precreated DBs & permissions](https://discuss.airbyte.io/t/4432)] - The DB is remote/slow or mis‑located, causing repeated timeouts. [[Temporal context deadline](https://github.com/airbytehq/airbyte/issues/59730); [AWS RDS timeouts](https://discuss.airbyte.io/t/2179)] ### What to check / do next Based on those patterns: 1. Verify external DB config (values.yaml & secrets) Ensure your Helm values for the external DB are correct (host, port, DB name, user, password, SSL options) and the secret keys match what the chart expects. [[External DB config](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database#values)] 2. Test connectivity from inside the cluster - From any pod with 
psql
available (or a debug pod), connect to the Temporal DB using the exact host/port/user used by Airbyte. - If you’re on Azure Postgres or similar managed services, confirm SSL requirements and certificates (the Azure case failed until SSL was configured correctly). [[Azure Postgres SSL](https://discuss.airbyte.io/t/2928)] 3. Look for root‑cause errors earlier in the same Temporal pod logs Before the 
database is closed
error, there is usually a more specific failure such as: - 
pq: database "temporal" does not exist
- 
permission denied
- 
connection refused
- 
read: connection timed out
Those are the kinds of messages that drove the “database is closed” state in the other threads. [[Unable to create SQL DB](https://discuss.airbyte.io/t/1544); [AWS RDS timeouts](https://discuss.airbyte.io/t/2179)] 4. Confirm the Temporal DB itself is reachable and healthy - DB instance up, accepting connections. - No firewall/VPC rule blocking pods. - If it’s cross‑region or high‑latency, consider placing DB and cluster in the same region (this resolved Temporal timeouts for one user). [[Temporal context deadline](https://github.com/airbytehq/airbyte/issues/59730)] 5. Restart Temporal after fixing DB issues Once connectivity and credentials are correct, restart the Temporal deployment so it rebuilds healthy connections to Postgres. Because the sources don’t document this specific “sql: database is closed” message or a precise fix for it, I can’t safely prescribe a one‑command solution. But all related documented cases were resolved by fixing external Postgres connectivity/permissions/SSL/latency, then restarting the Temporal pod.
f
@kapa.ai on server I have
Copy code
2025-12-15 09:18:44,723 [io-executor-thread-6]	ERROR	i.a.c.s.e.h.UncaughtExceptionHandler(handle):33 - Uncaught exception
io.micronaut.data.exceptions.DataAccessException: Error executing SQL Query: ERROR: relation "dataplane" does not exist
  Position: 203
	at io.micronaut.data.jdbc.operations.DefaultJdbcRepositoryOperations.findOne(DefaultJdbcRepositoryOperations.java:394)
	at io.micronaut.data.jdbc.operations.DefaultJdbcRepositoryOperations.lambda$findOne$0(DefaultJdbcRepositoryOperations.java:354)
	at io.micronaut.data.jdbc.operations.DefaultJdbcRepositoryOperations.lambda$executeRead$23(DefaultJdbcRepositoryOperations.java:819)
	at io.micronaut.data.connection.support.AbstractConnectionOperations.executeWithNewConnection(AbstractConnectionOperations.java:167)
	at io.micronaut.data.connection.support.AbstractConnectionOperations.execute(AbstractConnectionOperations.java:114)
	at io.micronaut.data.jdbc.operations.DefaultJdbcRepositoryOperations.executeRead(DefaultJdbcRepositoryOperations.java:816)
	at io.micronaut.data.jdbc.operations.DefaultJdbcRepositoryOperations.findOne(DefaultJdbcRepositoryOperations.java:354)
	at io.micronaut.data.runtime.intercept.DefaultFindOneInterceptor.intercept(DefaultFindOneInterceptor.java:45)
	at io.micronaut.data.runtime.intercept.DataIntroductionAdvice.intercept(DataIntroductionAdvice.java:84)
	at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:143)
	at io.airbyte.data.repositories.DataplaneRepository$Intercepted.findByServiceAccountId(Unknown Source)
	at io.airbyte.data.services.impls.data.DataplaneServiceDataImpl.getDataplaneByServiceAccountId(DataplaneServiceDataImpl.kt:129)
	at io.airbyte.server.services.DataplaneService.getDataplaneByServiceAccountId(DataplaneService.kt:79)
	at io.airbyte.server.apis.controllers.DataplaneController.initializeDataplane(DataplaneController.kt:174)
	at io.airbyte.server.apis.controllers.$DataplaneController$Definition$Exec.dispatch(Unknown Source)
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
	at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4438)
	at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272)
	at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
	at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:465)
	at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:442)
	at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92)
	at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.postgresql.util.PSQLException: ERROR: relation "dataplane" does not exist
  Position: 203
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2734)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2421)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:372)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:518)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:435)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:196)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:139)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
	at io.micronaut.data.jdbc.operations.DefaultJdbcRepositoryOperations.findOne(DefaultJdbcRepositoryOperations.java:361)
	... 25 common frames omitted

2025-12-15 09:18:48,328 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:18:54,362 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:19:00,395 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:19:06,426 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:19:12,453 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:19:18,479 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:19:24,508 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:19:30,532 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:19:36,556 [main]	INFO	o.f.c.i.l.s.Slf4jLog(info):41 - Schema history table "public"."airbyte_jobs_migrations" does not exist yet
2025-12-15 09:19:36,566 [main]	ERROR	i.m.r.Micronaut(handleStartupException):362 - Error starting Micronaut server: io.airbyte.db.check.DatabaseCheckException: Timeout while waiting for database to fulfill minimum flyway migration version..
java.lang.RuntimeException: io.airbyte.db.check.DatabaseCheckException: Timeout while waiting for database to fulfill minimum flyway migration version..
	at io.airbyte.server.DatabaseEventListener.onApplicationEvent(DatabaseEventListener.kt:37)
	at io.airbyte.server.DatabaseEventListener.onApplicationEvent(DatabaseEventListener.kt:20)
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:266)
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:226)
	at io.micronaut.http.server.netty.discovery.NettyServiceDiscovery.onStart(NettyServiceDiscovery.java:54)
	at io.micronaut.http.server.netty.discovery.$NettyServiceDiscovery$Definition$Exec.dispatch(Unknown Source)
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:456)
	at io.micronaut.context.DefaultBeanContext$BeanExecutionHandle.invoke(DefaultBeanContext.java:3987)
	at io.micronaut.aop.chain.AdapterIntroduction.intercept(AdapterIntroduction.java:91)
	at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:143)
	at io.micronaut.http.server.netty.discovery.NettyServiceDiscovery$ApplicationEventListener$onStart1$Intercepted.onApplicationEvent(Unknown Source)
	at io.micronaut.context.event.ApplicationEventPublisherFactory.notifyEventListeners(ApplicationEventPublisherFactory.java:266)
	at io.micronaut.context.event.ApplicationEventPublisherFactory$2.publishEvent(ApplicationEventPublisherFactory.java:226)
	at io.micronaut.http.server.netty.NettyHttpServer.fireStartupEvents(NettyHttpServer.java:720)
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:330)
	at io.micronaut.http.server.netty.NettyHttpServer.start(NettyHttpServer.java:124)
	at io.micronaut.runtime.Micronaut.start(Micronaut.java:81)
	at io.airbyte.server.ApplicationKt.main(Application.kt:17)
Caused by: io.airbyte.db.check.DatabaseCheckException: Timeout while waiting for database to fulfill minimum flyway migration version..
	at io.airbyte.db.check.DatabaseMigrationCheck$DefaultImpls.check(DatabaseMigrationCheck.kt:84)
	at io.airbyte.db.check.JobsDatabaseMigrationCheck.check(JobsDatabaseMigrationCheck.kt:17)
	at io.airbyte.server.DatabaseEventListener.onApplicationEvent(DatabaseEventListener.kt:35)
	... 17 common frames omitted
k
A message from kapa.ai
Your server logs show two key problems: 1. Queries failing with 
relation "dataplane" does not exist
2. Flyway never finishing: 
Schema history table "public"."airbyte_jobs_migrations" does not exist yet
followed by 
Timeout while waiting for database to fulfill minimum flyway migration version..
That means the Airbyte 2.0 jobs/config migrations are not being applied successfully to your Postgres, so required tables (like 
dataplane
and the Flyway schema history table) are missing. From the documentation and related issues: - Airbyte expects to create and own its schema in an empty database (or schema). With external Postgres you must: - Create the databases (
airbyte
, 
temporal
, 
temporal_visibility
) yourself. - Then let Airbyte/Temporal create all tables via migrations. [[External DB issue](https://github.com/airbytehq/airbyte/issues/49304)] - When this doesn’t complete, the logs loop on “schema history table … does not exist yet” and eventually time out, exactly as you’re seeing. [[Forum Flyway timeout](https://discuss.airbyte.io/t/2589)] Because you dropped the 
public
schema, you now have an empty DB again, but something is preventing the migrations from finishing. Given what’s in the docs and issues, here’s what you can safely do and check. --- ## 1. Ensure the DB is in a clean, supported state For an external database with Helm chart V2, the docs/issue show this pattern: 1. Create the databases (once) in Postgres (if not already done):
Copy code
sql    CREATE DATABASE airbyte WITH OWNER = <airbyte_user>;    CREATE DATABASE temporal WITH OWNER = <airbyte_user>;    CREATE DATABASE temporal_visibility WITH OWNER = <airbyte_user>;
(Owner/user must match what you configured in 
airbyte-config-secrets
/ 
global.database.*
.) [[External Postgres DBs](https://github.com/airbytehq/airbyte/issues/49304)] 2. Inside each DB, Airbyte/Temporal will create tables and Flyway schema tables automatically. You should not pre-create application tables like 
dataplane
yourself. Since you already dropped 
public
, it’s effectively clean. Just make sure: - You did this in the Airbyte config/jobs DB (whatever your `global.database.name`/`database` is set to). [[External DB config](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database#values)] - The user has full rights on that DB (create table, alter, etc.). Lack of permissions caused similar external-Postgres failures. [[SQL DB permissions](https://discuss.airbyte.io/t/1544)] --- ## 2. Re‑check your Helm values for external DB For Helm chart V2 with an external DB, 
values.yaml
should look like the documented example: [[External DB config](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database#values)]
Copy code
yaml postgresql:   enabled: false  global:   database:     type: external     secretName: airbyte-config-secrets   # or your secret     host: <your-db-host>     port: "<your-port>"     name: "<your-airbyte-db-name>"      # key is `name` in chart v2     userSecretKey: "database-user"     passwordSecretKey: "database-password"
And the secret:
Copy code
yaml apiVersion: v1 kind: Secret metadata:   name: airbyte-config-secrets type: Opaque stringData:   database-user: <user>   database-password: <password>
[[External DB secrets](https://docs.airbyte.com/platform/deploying-airbyte/integrations/database#secrets)] If any of these are misaligned (wrong DB name, wrong secret key names, etc.), migrations can fail or point at the wrong database. --- ## 3. Let Airbyte fully re‑initialize the DB Once the DB and values are correct: 1. (Recommended) Reinstall the Helm release to force a clean startup sequence, as shown in the official upgrade docs: [[How to upgrade](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#how-to-upgrade); [Deploy Airbyte](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#deploy-airbyte)]
Copy code
bash    helm uninstall airbyte -n airbyte     helm install airbyte airbyte-v2/airbyte \      --namespace airbyte \      --values ./values.yaml \      --version 2.0.18   # or the chart version you’re using
(Keep the namespace and reuse the same external DB.) 2. Watch the server and worker logs at startup for Flyway messages. Unlike your current logs, you should see Flyway creating tables and then stopping the “schema history table does not exist yet” loop once the first migration succeeds. If, after a clean reinstall with correct DB settings, you still see: - Repeated 
Schema history table "public"."airbyte_jobs_migrations" does not exist yet
- Followed by 
Timeout while waiting for database to fulfill minimum flyway migration version..
then, based on the sources, the remaining likely causes are: - Permission problems on the DB user (can’t create the Flyway tables or other objects) [[SQL DB permissions](https://discuss.airbyte.io/t/1544)] - Connectivity/latency problems that make migrations time out before finishing, similar to earlier Temporal issues with remote RDS/Azure Postgres. [[Temporal context deadline](https://github.com/airbytehq/airbyte/issues/59730); [AWS RDS timeouts](https://discuss.airbyte.io/t/2179)] --- ## 4. About the 
dataplane
table error 
ERROR: relation "dataplane" does not exist
is a symptom of the jobs/config migrations not having run to completion (that table is one of the objects they should create). Once Flyway succeeds in setting up the schema, that error should disappear. The knowledge sources you provided do not document a manual SQL script or supported manual creation of the 
dataplane
table, so I cannot recommend creating it yourself. The supported path is to: - Start from an empty schema/DB, with correct external DB configuration, - Let Airbyte 2.0’s migrations create all tables automatically. --- ### If it still fails The docs and issues you shared do not contain a deeper troubleshooting flow for Flyway migration timeouts in Airbyte 2.0 (no specific SQL to run, nor a dedicated repair tool). If, after: - ensuring a clean DB, - verifying values/secrets and permissions, - and doing a clean Helm uninstall/install, you still hit the same Flyway timeout, that’s beyond what’s covered in these sources. In that case, you’d need to open an issue with: - Exact Airbyte version and chart version, - Full server logs from startup (including all Flyway lines), - Your 
global.database.*
configuration (without passwords). From the documentation, though, the next concrete steps are: 1. Verify DB/secret config matches the external DB docs. 2. Ensure the DB is empty and permissions are sufficient. 3. Reinstall the Helm release and let Airbyte initialize the schema.
2 Views
Open in Slack
PreviousNext
Powered by