@kapa.ai How does PyAirbyte handle incremental syncs and how is the state taken care of?

PyAirbyte Incremental Syncs & State Management PyAirbyte handles incremental syncs through its

write()

method on the

Destination

class, with state managed via a cache mechanism. How Incremental Syncs Work When writing data to a destination, PyAirbyte follows this priority order for determining the sync starting point: 1. Cache/state cache: If a

cache

or

state_cache

has tracked state, that state is used for the sync. 2. Destination state: If no cache state is available but a known destination state exists, that is used. 3. Full refresh fallback: If neither is available, a full refresh is performed. [Destination write method] State Management Parameters The

write()

method provides several parameters to control state behavior:

destination.write(
    source_data=source,
    streams=["my_stream"],
    cache=my_cache,           # Used for reading source data AND tracking incremental state
    state_cache=my_state_cache,  # Optional: separate cache just for storing incremental state
    write_strategy=WriteStrategy.AUTO,
    force_full_refresh=False  # Set True to ignore existing state and reload all data
)

•

cache

: If provided, it is used both for reading source data and for tracking incremental state. You don't need to set

state_cache

separately if

cache

is specified. •

state_cache

: A dedicated cache for storing incremental state, useful when

cache

is not set or when

source_data

is already a

ReadResult

. •

force_full_refresh

: Setting this to

True

ignores any existing state and reloads all source data from scratch. •

state_cache=False

: Disables state management entirely. [Destination write method] Underlying Protocol At the protocol level, Airbyte sources emit

AirbyteStateMessage

objects during a sync to checkpoint progress. This allows a subsequent sync to resume from where the last one left off — even in the case of partial failures. The state acts as a "high watermark" cursor (e.g., an

updated_at

timestamp), and only records newer than that cursor are fetched in the next sync. [State & Checkpointing] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai What do the values of

cache

and

state_cache

look like?

Based on the knowledge sources, here's what

cache

and

state_cache

values look like: cache The

cache

parameter accepts a

CacheBase

instance (or

False

to disable caching, or

None

for no cache). The available concrete cache implementations are: •

DuckDBCache

– local DuckDB database (default) •

PostgresCache

•

BigQueryCache

•

SnowflakeCache

•

MotherDuckCache

•

GenericSQLCacheConfig

[Caches module] You can get a cache instance using helper functions:

import airbyte as ab

# Default local DuckDB cache
cache = ab.get_default_cache()

# Or a new local DuckDB cache
from airbyte.caches import new_local_cache
cache = new_local_cache(cache_name="my_cache", cache_dir="/tmp/caches")

# Or a specific cache type, e.g. Postgres
from airbyte.caches import PostgresCache
cache = PostgresCache(
    host="localhost",
    port=5432,
    database="mydb",
    username="user",
    password="password",
    schema_name="public",
)

[Caches util] state_cache The

state_cache

parameter also accepts a

CacheBase

instance (or

False

to disable state management). It is the same type as

cache

— any of the SQL cache implementations listed above. The difference is its purpose: it is used only for storing incremental state, not for buffering source data. From the docs: > You do not need to set this if

cache

is specified or if

source_data

is a

Cache

object. Set to

False

to disable state management. [Destination write method] So in practice, if you already provide a

cache

, you typically don't need to set

state_cache

separately — the cache handles both data buffering and state tracking. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.