@kapa.ai a replication-job pod has a

NotReady

status, and checking the logs I can see a WARN after an exception:

2026-10-13 03:15:33,202 [pool-4-thread-4]       ←[31mWARN←[0;39m        i.a.w.i.VersionedAirbyteStreamFactory(internalLog$io_airbyte_airbyte_commons_worker):245 - WARN DefaultDispatcher-worker-1 c.c.c.a.Client(lambda$query$5):1582 Retrying. org.apache.hc.core5.http.NoHttpResponseException: dz8opq8f08.eu-west-1.vpce.aws.clickhouse.cloud:8443 failed to respond                                                                                                                          at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:301) ~[httpcore5-5.2.4.jar:5.2.4]

Thinking completed ClickHouse Connection Issue: NoHttpResponseException The error you're seeing indicates that Airbyte's replication pod is unable to get a response from your ClickHouse Cloud endpoint (

dz8opq8f08.eu-west-1.vpce.aws.clickhouse.cloud:8443

). This is a network connectivity issue between the Airbyte worker and ClickHouse. Likely Causes & Steps to Investigate 1. Network / Firewall / VPC Endpoint Access Since the hostname contains

vpce.aws

(a VPC endpoint), this is a private endpoint configuration. Ensure that: • Your Airbyte host's IP is allowlisted in your ClickHouse Cloud settings. • The VPC endpoint is correctly configured and the Airbyte pod can route traffic to it on port 8443 (HTTPS). • Security groups and firewall rules permit inbound connections from your Airbyte host to the ClickHouse endpoint. [Setup Guide] 2. Hostname Configuration Make sure the hostname in your connector configuration does not include the protocol prefix (e.g.,

https://

or

http://

). The connector explicitly rejects hostnames starting with `http`:

Please remove the protocol (http://, https://) from the hostname in your connector configuration.

[ClickhouseChecker.kt] 3. SSH Tunneling as a Fallback If the ClickHouse endpoint cannot be made directly accessible from the Airbyte pod (e.g., due to VPC restrictions), consider using SSH tunneling via a bastion host that has access to the ClickHouse instance. [Setup Guide] 4. ClickHouse Version Ensure your ClickHouse instance is running version 21.8.10.19 or later (tested against 26.7). [Requirements] Summary The

NoHttpResponseException

on port 8443 strongly points to a network-level failure — the TCP connection is established but ClickHouse (or an intermediate proxy/VPC endpoint) is not returning an HTTP response. Start by verifying that the Airbyte pod's egress traffic can reach the VPC endpoint on port 8443 without being blocked by security groups or NACLs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai all the other jobs succeeded, and this is the only job that is hanging. What can I do further in K8s or Airbyte to solve this situation

Thinking completed I'm sorry, but the knowledge sources do not contain enough information to provide specific guidance on resolving a single hanging replication pod in Kubernetes while other jobs succeed. However, based on what is available in the knowledge sources, here are a couple of relevant suggestions: 1. Cancel the Hanging Job and Let It Retry From a similar reported issue, manually cancelling a hanging sync can "unblock" it and allow a new attempt to take over. [Forum Post] 2. Set a Sync Timeout To avoid having to manually cancel in the future, you can configure the

SYNC_JOB_MAX_TIMEOUT_DAYS

environment variable so that stuck jobs cancel themselves automatically:

SYNC_JOB_MAX_TIMEOUT_DAYS=1

Note that the minimum value is 1 day. [Forum Post] 3. Inspect the Pod Directly You can inspect the hanging pod's state in Kubernetes for more clues:

# Check events for failures
kubectl get events --namespace <airbyte-namespace> --field-selector reason=FailedCreate

# Check node resource allocation
kubectl describe nodes | grep -A 5 "Allocated resources"

# Check workload-launcher logs
kubectl logs -l app.kubernetes.io/name=workload-launcher --namespace <airbyte-namespace> --tail=100

[Github Issue] For deeper diagnosis of why specifically the ClickHouse replication pod is hanging while others succeed, I'd recommend opening a GitHub issue or a forum post with the full pod logs, as the knowledge sources don't cover this specific scenario in detail. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I see many of these kind of exceptions on clickhouse side:

Code: 60. DB::Exception: Unknown table expression identifier 'GL_Warehouse_01Training_AppLogging.GL_WarehpLoggingBatteryEvent8ad8656a0b14653cc48585891cf96a43' in scope SELECT count(1) AS cnt FROM GL_Warehouse_01Training_AppLogging.GL_WarehpLoggingBatteryEvent8ad8656a0b14653cc48585891cf96a43. (UNKNOWN_TABLE) (version 26.4.1.2596 (official build))

Thinking completed This is directly related to a known issue in the ClickHouse destination connector. The

UNKNOWN_TABLE

error is being thrown because Airbyte is issuing a

COUNT

query against a temporary table that no longer exists (it was already cleaned up after a successful sync). Root Cause The direct-load status gatherer probes generated temporary tables on every sync. After a successful cleanup, those temp tables are absent — but the connector was previously issuing a

COUNT

query anyway, causing ClickHouse to throw

UNKNOWN_TABLE

. [PR #82684] The fix was to check table existence before counting it:

override suspend fun countTable(tableName: TableName): Long? {
    try {
        // The direct-load status gatherer probes generated temporary tables on every
        // sync. Those tables are normally absent after a successful cleanup, so avoid
        // issuing a failing COUNT query just to discover that expected state.
        if (!tableExists(tableName)) {
            return null
        }
        val sql = sqlGenerator.countTable(tableName, "cnt")
        ...
    } catch (_: Exception) {
        return null
    }
}

[ClickhouseAirbyteClient.kt] What You Should Do 1. Upgrade your ClickHouse destination connector to the version that includes PR #82684. This fix specifically targets the

UNKNOWN_TABLE

errors you're seeing and prevents the failing

COUNT

queries from being emitted at all. 2. In the meantime, these errors are not data-loss errors — they occur during the cleanup/status-check phase for temporary tables that are expected to be absent after a successful sync. Your actual data sync should be unaffected. 3. The hanging pod you observed earlier may be related — if the connector is repeatedly retrying against a missing temp table and hitting errors, it could contribute to the job stalling. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai got another exception after retrying:

io.airbyte.api.client.ApiException: Gone                                                                                                                                                                                                             at io.airbyte.workload.api.client.WorkloadApiClient.unit(WorkloadApiClient.kt:405)                                                                                                                                                           at io.airbyte.workload.api.client.WorkloadApiClient.workloadHeartbeat(WorkloadApiClient.kt:66)                                                                                                                                               at io.airbyte.container.orchestrator.worker.WorkloadHeartbeatSender.sendHeartbeat(WorkloadHeartbeatSender.kt:84)                                                                                                                             at io.airbyte.container.orchestrator.worker.WorkloadHeartbeatSender$sendHeartbeat$1.invokeSuspend(WorkloadHeartbeatSender.kt)                                                                                                                at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)                                                                                                                                                    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)                                                                                                                                                                              at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)                                                                                                                                                     at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)                                                                                                                                            at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)                                                                                                                                              at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

Thinking completed It looks like your message was cut off — could you please share the full exception text? That way I can help diagnose what happened after the retry. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai

io.airbyte.api.client.ApiException: Gone                                                                                                                                                                                                             at io.airbyte.workload.api.client.WorkloadApiClient.unit(WorkloadApiClient.kt:405)                                                                                                                                                           at io.airbyte.workload.api.client.WorkloadApiClient.workloadHeartbeat(WorkloadApiClient.kt:66)                                                                                                                                               at io.airbyte.container.orchestrator.worker.WorkloadHeartbeatSender.sendHeartbeat(WorkloadHeartbeatSender.kt:84)                                                                                                                             at io.airbyte.container.orchestrator.worker.WorkloadHeartbeatSender$sendHeartbeat$1.invokeSuspend(WorkloadHeartbeatSender.kt)                                                                                                                at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)                                                                                                                                                    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)                                                                                                                                                                              at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:586)                                                                                                                                                     at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:829)                                                                                                                                            at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:717)                                                                                                                                              at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:704)

Thinking completed This exception is a

410 Gone

response from the Workload API during a heartbeat, which means the Workload Monitor already marked the workload as terminated (timed out or failed) before the pod finished its work. This matches exactly the pattern described in the knowledge sources. What Happened The sequence was likely: 1. The pod was hanging (due to the ClickHouse

UNKNOWN_TABLE

/

NoHttpResponseException

issues you saw earlier). 2. The Workload Monitor detected that the heartbeat deadline had expired and automatically failed the workload. 3. When the pod eventually tried to send a heartbeat, the workload was already in a terminal state — hence

410 Gone

. [Workload Monitor] This is also consistent with the earlier proxy issue report, where the same

410 Gone

/ terminal state error was seen:

io.airbyte.workload.api.client.generated.infrastructure.ClientException: 
Client error : 410 Gone {"message":"Heartbeat a workload in a terminal state"}

[Github Issue] What To Do Next 1. Check the Workload Monitor failure message in the Airbyte UI job history — it will tell you which check triggered the failure (

Not heartbeating

,

Timeout

, etc.). 2. *Inspect the pod logs before it was killed:*kubectl logs replication-pod-name -n namespace -c main --previous 3. *Check for OOMKill or crash:*kubectl describe pod replication-pod-name -n namespaceLook for

OOMKilled

,

CrashLoopBackOff

, or resource pressure events. [Workload Monitor] 4. Most importantly — upgrade the ClickHouse destination connector to get the

UNKNOWN_TABLE

fix. The repeated ClickHouse exceptions are likely what caused the pod to stall long enough for the Workload Monitor to time it out, triggering this cascade. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.