https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> How does the Airbyte heartbeat proc...
# replication-ask-ai
j
@kapa.ai How does the Airbyte heartbeat process works?
k
Airbyte uses a heartbeat mechanism to detect unresponsive sources and destinations and automatically restart stuck syncs. At a high level: - Airbyte watches for “heartbeat” activity during a sync. - If no heartbeat is seen within a configured time window, the job is failed with an error like “The source is unresponsive” or “The destination is unresponsive,” and Airbyte restarts the job. [[Heartbeats](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats)] ### Source heartbeats - Any 
RECORD
or 
STATE
message emitted by the source counts as a heartbeat. [[Heartbeats tech](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] - A background process tracks the time since the last such message. - If the time exceeds a threshold, the sync attempt is failed with cause “The source is unresponsive” (internally marked as a heartbeat timeout). [[Heartbeats tech](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] Configuration (in 
flags.yaml
): [[Heartbeats tech](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] - `heartbeat-max-seconds-between-messages`: max allowed time between two source messages (default: 3 hours). - `heartbeat.failSync`: - 
true
(default): a missed heartbeat fails the sync. - `false`: missed heartbeat is logged but does not fail the sync. ### Destination “heartbeat” / timeout Destinations don’t continuously send messages back, so Airbyte uses a timeout-style check instead: [[Heartbeats tech](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] - The platform monitors each call to the destination. - If a call runs longer than a configured limit, the destination is considered timed out and the sync can be failed with “The destination is unresponsive.” Configuration (in 
flags.yaml
): [[Heartbeats tech](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] - `destination-timeout-max-seconds`: max allowed duration of a single destination call (default: 24 hours). - `destination-timeout.failSync`: - 
true
(default): destination timeout fails the sync. - `false`: timeout is logged but sync continues. This mechanism exists as a “last resort” safety net to avoid jobs hanging indefinitely and to surface potential issues with sources, destinations, or infrastructure. [[Heartbeats](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats)]
j
@kapa.ai What are the common reasons for timeout in hearbeat for the Postgres source?
k
A message from kapa.ai
The knowledge sources don’t list Postgres‑specific heartbeat timeouts as a separate category, but they do describe the general heartbeat mechanism and common causes that apply to database sources like Postgres. For a Postgres source, the most common reasons a heartbeat/timeout error occurs are: 1. Slow or long‑running queries on Postgres Database sources can be slow to respond due to: database size, complex queries, or many concurrent queries. If a sync query takes too long, the source won’t emit 
RECORD
or 
STATE
messages for an extended period, so Airbyte’s heartbeat monitor concludes “The source is unresponsive.” [[Heartbeats](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#known-heartbeat-error-causes)] - A very common pattern is using an unindexed cursor column for incremental syncs, which makes queries scan large tables and respond slowly. [[Heartbeats](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#known-heartbeat-error-causes)] 2. Under‑provisioned Postgres instance If the Postgres server is short on CPU, memory, or I/O, queries from Airbyte can become very slow, again delaying messages long enough to trigger a heartbeat timeout. [[Heartbeats](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#known-heartbeat-error-causes)] 3. General connectivity / resource issues on the source DB The docs note that database source heartbeat errors are rare and usually indicate a real issue that needs investigation (e.g., load spikes, network hiccups, or DB configuration problems). [[Heartbeats](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#known-heartbeat-error-causes)] 4. Heartbeat configuration too strict (less common) Airbyte fails a sync if the time between source messages exceeds 
heartbeat-max-seconds-between-messages
(default 3 hours). If this is set too low relative to how long your Postgres queries naturally take, you can see heartbeat timeouts even though the DB is technically still working. [[Heartbeats tech](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)] There is no Postgres‑specific heartbeat setting in the connector itself in the provided docs; all heartbeat behavior is controlled at the platform level via 
flags.yaml
. [[Heartbeats tech](https://docs.airbyte.com/platform/understanding-airbyte/heartbeats#technical-details)]
2 Views
Open in Slack
PreviousNext
Powered by