@kapa.ai airbyte is running a

Select * from table

running a full refresh on a table without using partitions/primary keys filtering that it's faster. Why?

Airbyte does not simply run a plain

SELECT * FROM table

for full refresh syncs on large tables. Instead, it uses a chunked/partitioned approach for performance and resumability reasons. Why Airbyte Uses Partitioned Queries Instead of a Simple SELECT * A naive

SELECT * FROM table

query would cause several problems: 1. Long-lasting transaction locks on the source table, slowing it down. 2. Memory issues — large tables may not fit in memory, potentially crashing the database. 3. No resumability — if the sync fails mid-way, there's no cursor to resume from. To avoid these issues, Airbyte breaks the read into chunks. For example, with a Postgres source it issues queries like:

SELECT * FROM users WHERE CTID >= X AND CTID < Y
SELECT * FROM users WHERE CTID >= Y AND CTID < Z
...

This pagination approach keeps the database healthy and allows Airbyte to resume a failed sync from where it left off. [Resumability docs] The Trade-off: At-Least-Once Delivery Because of this chunked approach, Airbyte may deliver a record more than once (it is an at-least-once system). If a sync fails partway through a chunk, that chunk's records will be re-sent on the next attempt. If deduplication is required, you should choose a sync mode that includes deduplication (e.g., Full Refresh Overwrite + Deduped). [Resumability docs] Known Issue: Sampling Queries Without Partition Bounds There is a known bug specifically in the source-mssql connector where the sampling query used to determine partition split boundaries runs against the entire table without filtering by partition bounds. This can cause: • Duplicate records (~2x–10x) in Full Refresh Append syncs on large tables. • Significantly increased sync time and resource usage. A fix has been submitted (PR #74756) that passes partition bounds into the sampling query. [MSSQL duplicate issue; fix PR] If you are seeing a plain unfiltered

SELECT TOP 1024 ... ORDER BY [column]

query in MSSQL, that is the sampling query used to determine split boundaries — and it is currently running without a WHERE clause filter due to this bug. [MSSQL sampling issue] In summary: Airbyte intentionally avoids a single

SELECT *

in favor of chunked reads for resilience and performance. If you're seeing unfiltered queries, it may be related to the known MSSQL sampling bug. Could you share which connector/source you're using? That would help narrow down the specific behavior you're observing. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I'm speaking about mysql source connector that is running a

select * from table

instead of dividing it in chunks

Based on the knowledge sources available, I don't have specific documentation explaining why the MySQL source connector would run a plain

SELECT * FROM table

without chunking. However, there are a few relevant points from the knowledge sources: Possible Causes 1. Table size threshold: The MySQL connector uses concurrent/partitioned reads for large tables. The partitioning logic in

MySqlSourceJdbcPartitionFactory

is triggered based on table size (e.g., tables > 4GB). If your table is below that threshold, it may not trigger the chunked read path. [ClassCastException issue] 2. Known bugs in partition boundary calculation: There have been reported bugs in

MySqlSourceJdbcPartitionFactory

(versions 3.50.x) where the partition boundary calculation fails with errors like

ArithmeticException: / by zero

, causing syncs to fall back or fail entirely on large tables. [ArithmeticException issue] 3. Performance issues with sampling queries: In connector versions around 3.11.x, a random row sampling query was introduced to determine partition split boundaries, which caused severe performance degradation on large tables. This was reportedly fixed in version 3.11.8/3.11.9. [sampling performance issue] What to Check • Connector version: Make sure you're on a recent version of

source-mysql

(the fix for partition boundary issues was released in

v3.0.8

for the ArithmeticException bug). [ArithmeticException fix] • Table size: If the table is small enough, Airbyte may intentionally read it in a single query. • Primary key: The concurrent partitioned read requires a primary key. Tables without primary keys may be read differently. [MySQL troubleshooting] The knowledge sources don't explicitly document the exact size threshold that triggers chunked reads for MySQL, so I'd recommend checking your connector version and whether your table has a primary key as a starting point. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai is there any way to instruct airbyte to give a failure on a sync job after 1 hour still running?

Yes, Airbyte provides several timeout mechanisms you can configure: 1. Overall Job Timeout (SYNC_JOB_MAX_TIMEOUT_DAYS) This env variable sets the maximum number of days a sync job can run before timing out. The default is 3 days. Note that the minimum granularity is 1 day, so you cannot set it to 1 hour using this variable. [Configuring Airbyte] 2. Source Heartbeat Timeout If your source stops emitting

RECORD

or

STATE

messages, the platform will fail the sync with

The source is unresponsive

. This is configured via `flags.yaml`: • `heartbeat-max-seconds-between-messages`: Maximum time allowed between 2 messages. Default is 3 hours. • `heartbeat.failSync`: Set to

true

(default) to fail the sync on a missed heartbeat. So you could set

heartbeat-max-seconds-between-messages: 3600

(3600 seconds = 1 hour) to fail the sync if the source goes silent for 1 hour. [Heartbeats] 3. Destination Timeout For the destination side, configure via `flags.yaml`: • `destination-timeout-max-seconds`: Fails the sync if a call to the destination exceeds this duration. Default is 24 hours. • `destination-timeout.failSync`: Set to

true

(default) to fail on timeout. You could set

destination-timeout-max-seconds: 3600

to enforce a 1-hour timeout on destination calls. [Heartbeats] Summary: There is no single setting to hard-stop a sync after exactly 1 hour of total runtime, but you can combine the source heartbeat and destination timeout settings (both configurable to 3600 seconds) to effectively fail a sync that appears stuck or unresponsive within 1 hour. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai where do I have to set`heartbeat-max-seconds-between-messages ?`

You need to set

heartbeat-max-seconds-between-messages

in the

flags.yaml

file of your Airbyte deployment. [Heartbeats] Here's an example of what the relevant section in

flags.yaml

would look like:

heartbeat-max-seconds-between-messages: 3600
heartbeat.failSync: true

The knowledge sources don't provide the exact file path or location of

flags.yaml

within the deployment, so I'd recommend checking your Airbyte Helm chart or deployment configuration for where this file is mounted or referenced. If you're using a Kubernetes/Helm deployment, it may be configurable via

values.yaml

as an environment variable or config map entry. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai where do I have to put flags.yaml file?

Based on the knowledge sources available, there is a GitHub issue that provides relevant context about how

flags.yaml

is used in a Kubernetes/Helm deployment: The

flags.yaml

file is mounted as a ConfigMap into the Airbyte pods. In a Helm-based deployment, you would: 1. Create a Kubernetes ConfigMap containing your

flags.yaml

content. 2. Mount it into the relevant pods using

global.extraVolumes

and

global.extraVolumeMounts

in your

values.yaml

. For example, in your `values.yaml`:

global:
  extraVolumes:
    - name: flags-config
      configMap:
        name: airbyte-flags
  extraVolumeMounts:
    - name: flags-config
      mountPath: /app/flags.yaml
      subPath: flags.yaml

[global.extraVolumes issue] > ⚠️ Important caveat: There is a known bug in Helm Chart V2 (2.0.19) where

global.extraVolumes

is missing from the

server

,

connector-builder-server

, and

temporal

deployment templates, causing the volume mount to silently fail. The workaround is to patch those templates locally or use component-level

extraVolumes

(e.g.,

worker.extraVolumes

) instead. [global.extraVolumes issue] The knowledge sources do not explicitly document the exact expected mount path for

flags.yaml

inside the containers, so I'd recommend checking your specific Airbyte version's documentation or the Helm chart source for the precise path. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai instead of using flags.yaml could I put it on values.yaml?

The knowledge sources do not explicitly document a way to set

heartbeat-max-seconds-between-messages

directly in

values.yaml

as a named parameter. However, based on the knowledge sources, Airbyte's

values.yaml

supports an

env_vars

section for various components, which allows you to pass environment variables to pods. If the heartbeat settings are exposed as environment variables, you could potentially set them there. For example:

worker:
  env_vars:
    HEARTBEAT_MAX_SECONDS_BETWEEN_MESSAGES: 3600

That said, the knowledge sources do not explicitly confirm that

heartbeat-max-seconds-between-messages

from

flags.yaml

has a direct environment variable equivalent that can be set in

values.yaml

. The documentation consistently refers to

flags.yaml

as the configuration mechanism for heartbeat settings. [Heartbeats] I'd recommend checking the Airbyte Helm chart values reference [Values.yaml reference] or the Airbyte GitHub repository for the full list of supported

env_vars

to confirm whether this specific setting is available as an environment variable override. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.