<@U04S8PLJWMT> where should one configure heart be...
# ask-ai
a
@kapa.ai where should one configure heart beat timeout? We have a long running sync, eventhough sync is running Airbyte has marked satus as success
k
Airbyte has two different timeout/heartbeat mechanisms, each configured in a different place: --- ### 1. Global job timeout (job marked success while still “running” in reality) This is the overall Temporal job timeout. It’s controlled via the
SYNC_JOB_MAX_TIMEOUT_DAYS
environment variable on the Airbyte platform. [[Sync hangs post](https://discuss.airbyte.io/t/709)] Example in
.env
(or equivalent env vars in your deployment):
Copy code
bash SYNC_JOB_MAX_TIMEOUT_DAYS=3   # minimum is 1 day
This affects how long a sync attempt may run before Airbyte times it out at the workflow level. If this is too large, a long‑running sync may effectively “outlive” other checks and you might see confusing states. --- ### 2. Heartbeat / unresponsive Source or Destination These are configured in
flags.yaml
on the Airbyte platform, not on the connection or connector itself. [[Heartbeats technical](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] Source heartbeat (no RECORD/STATE messages for too long):
Copy code
yaml heartbeat-max-seconds-between-messages: 10800   # default 3h heartbeat.failSync: true                        # fail sync on missed heartbeat
Destination timeout (destination call taking too long):
Copy code
yaml destination-timeout-max-seconds: 86400   # default 24h destination-timeout.failSync: true       # fail sync on destination timeout
The monitoring process uses these values to decide when to fail a sync with messages like “The source is unresponsive” or “The destination is unresponsive.” [[Heartbeats technical](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] --- ### How this relates to your case - If Airbyte is marking the sync as success even though, from your perspective, it’s still “running” downstream (e.g., external system still working), that is usually governed by: - The connector’s own logic (e.g., async API polling with a “Polling Job Timeout” in Connector Builder), or - The global job timeout (
SYNC_JOB_MAX_TIMEOUT_DAYS
) and heartbeats/flags above. - For async API connectors built with Connector Builder UI, there is also a “Polling Job Timeout” on the Polling tab that defines how long to wait for a remote job before timing out that stream. [[Polling tab](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/async-streams#polling-tab)] If you share whether this is a DB sync vs. an async API connector, and how you deploy Airbyte (Docker / K8s / Cloud), I can point to the exact file/env to edit for your setup.
a
@kapa.ai deployment in K8s
k
A message from kapa.ai
On Kubernetes, the heartbeat/timeout behavior you asked about is controlled at the platform level, not per‑connection, and is configured via: 1. Environment variables on the Airbyte platform pods (for overall job timeouts), and 2. A
flags.yaml
file mounted into the platform
(for source/destination heartbeats). --- ### 1. Overall sync job timeout (env var) This controls how long a sync job is allowed to run before Airbyte times it out at the workflow level:
Copy code
bash SYNC_JOB_MAX_TIMEOUT_DAYS=3
This is documented as “Defines the number of days a sync job will execute for before timing out.” [[Configuring jobs](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#jobs)] In K8s, you set this as an environment variable on the Airbyte Server/Worker (depending on your chart or manifests). For Helm, it’s usually exposed via values that map to env vars on those pods. --- ### 2. Heartbeat / timeout for source and destination (flags.yaml) The logic that marks a source/destination as unresponsive and fails the sync is configured in a
flags.yaml
file
loaded by the platform. [[Heartbeats technical](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] Relevant keys:
Copy code
yaml # Source heartbeat (no RECORD/STATE messages for too long) heartbeat-max-seconds-between-messages: 10800   # default 3 hours heartbeat.failSync: true                        # fail sync if missed heartbeat  # Destination timeout (single destination call taking too long) destination-timeout-max-seconds: 86400          # default 24 hours destination-timeout.failSync: true              # fail sync if destination times out
On Kubernetes, you typically: 1. Create a ConfigMap containing
flags.yaml
. 2. Mount it into the Airbyte platform pod(s) (Server/Worker) at the path they expect for
flags.yaml
. The docs explicitly state that these heartbeats are configured using
flags.yaml
and that they are platform‑wide, not per‑connection. [[Heartbeats technical](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] --- If your long‑running sync is being marked success even though you think it should still be “running”, the relevant knobs to tighten in K8s are: - Lower
destination-timeout-max-seconds
if the destination can hang for a long time. - Lower
heartbeat-max-seconds-between-messages
if the source is quiet for too long. - Adjust
SYNC_JOB_MAX_TIMEOUT_DAYS
if the overall job window is too generous. [[Configuring jobs](https://docs.airbyte.com/platform/operator-guides/configuring-airbyte#jobs); [Heartbeats technical](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)]