We are running *Airbyte OSS 1.8.3* installed via `...
# replication-ask-ai
b
We are running Airbyte OSS 1.8.3 installed via
abctl
on a single-node Kubernetes cluster (kind). We are seeing sync jobs fail with:
termination_source: workload-monitor-start
termination_reason: "Airbyte could not start the process within time limit. The workload was claimed but never started."
The workload
status
column in Postgres transitions
pending -> claimed -> (never reaches running) -> failure
. The
deadline
column in the
workload
table is NULL for all rows. The time between
created_at
and
updated_at
on the failed rows is consistently ~30 minutes, which tells us there is a hardcoded 30-minute timeout on the
claimed
state. What we have checked and ruled out: We dumped every environment variable from all three relevant pods (
airbyte-abctl-worker
,
airbyte-abctl-workload-launcher
,
airbyte-abctl-workload-api-server
) and searched the
airbyte-abctl-airbyte-env
configmap. There is no existing env var containing the words TIMEOUT, DEADLINE, MONITOR, CLAIM, or DURATION in any of those pods. The launcher does have
RUNNING_TTL_MINUTES=
(empty),
SUCCEEDED_TTL_MINUTES=10
, and
UNSUCCESSFUL_TTL_MINUTES=120
, but those are row-cleanup TTLs, not the start-phase timeout. What we need: In Airbyte OSS 1.8.3, what is the environment variable or configmap key that controls how long a workload is allowed to stay in the
claimed
state before
workload-monitor-start
kills it with "The workload was claimed but never started"? We want to raise it from 30 minutes to 2 hours. If the threshold is hardcoded and not configurable in 1.8.3, is there a later version where it becomes configurable, and if so, what is the key name?
k
The knowledge sources provided do not contain enough information to definitively answer your specific question about the environment variable controlling the "claimed but never started" timeout in Airbyte OSS 1.8.3. Here is what the knowledge sources do reveal that is relevant: What the logs show about WorkloadMonitor timeouts From
airbyte-cron
logs visible in multiple issues, the
WorkloadMonitor
runs several distinct checks: •
cancelNotStartedWorkloads
— checks for workloads that were never started •
cancelNotClaimedWorkloads
— checks for workloads that were never claimed •
cancelNotHeartbeatingWorkloads
— checks for non-heartbeating workloads •
cancelRunningForTooLongNonSyncWorkloads
— timeout value
PT4H
•
cancelRunningForTooLongSyncWorkloads
— timeout value
PT720H
(30 days) [sync jobs hanging][temporal deadline exceeded] The
cancelNotStartedWorkloads
method (which corresponds to your
workload-monitor-start
termination source) is a separate check from the running-too-long checks, and its timeout value is not shown in any of the log lines in the knowledge sources. What the knowledge sources do NOT cover The knowledge sources do not document: • The specific environment variable or configmap key controlling the
claimed → not started
timeout (the ~30-minute threshold you are observing) • Whether this threshold is hardcoded or configurable in version 1.8.3 • A version in which it becomes configurable What is documented as configurable The officially documented job-related timeout variable is
SYNC_JOB_MAX_TIMEOUT_DAYS
, which controls how long a sync job runs before timing out overall — but this is distinct from the start-phase timeout you are hitting. [configuring Airbyte] There is also a known issue where
SYNC_JOB_MAX_TIMEOUT_DAYS
set on the
worker
pod had no effect, suggesting the timeout enforcement was moved to `airbyte-cron`'s
WorkloadMonitor
. [SYNC_JOB_MAX_TIMEOUT_DAYS issue] Recommendation Given the limitations of the available knowledge sources, I'd recommend: 1. Checking the
airbyte-cron
pod's environment variables directly, as
WorkloadMonitor
runs there and may read a configurable threshold for the
cancelNotStartedWorkloads
check. 2. Filing or searching a GitHub issue on the airbytehq/airbyte-platform repository specifically for the
cancelNotStartedWorkloads
timeout configuration, as the knowledge sources do not contain enough detail to confirm whether it is configurable in 1.8.3. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
b
@kapa.ai What about the latest 2.1.0 airbyte version - any
cancelNotStartedWorkloads
or similar checks we could set, or is this simply something we cannot control?
k
The knowledge sources do not contain any information about configurable environment variables or settings for
cancelNotStartedWorkloads
(or similar claimed-state timeout checks) in Airbyte 2.1.0. What the 2.1.0 release notes do document is a new configurable option related to the Workload API host: [v2.1 release notes] >
WORKLOAD_API_HOST
is now configurable
: You can now override the
WORKLOAD_API_HOST
value in your Helm values.yaml, enabling custom workload API host configurations in self-managed deployments. However, this is unrelated to the start-phase timeout you are hitting. The Helm chart V2 migration guide lists
SYNC_JOB_MAX_TIMEOUT_DAYS
as still being set via
extraEnvs
(no dedicated Helm value), but again this controls overall sync job duration, not the claimed-but-never-started window. [chart V2 upgrade] In short: The knowledge sources do not provide evidence that the
cancelNotStartedWorkloads
timeout (the ~30-minute claimed-state deadline) is configurable in Airbyte 2.1.0, nor do they document a specific environment variable for it in any version. I cannot confirm whether it remains hardcoded or has become configurable in 2.1.0 based on the available information. Your best path forward would be to search or file an issue on the airbytehq/airbyte-platform repository targeting this specific check in
WorkloadMonitor
. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.