I want to use pyairbyte over using airbyte self hosted. But can I somehow run pyairbyte against destinations or are they incompatible? Do they use different state?

Yes, PyAirbyte supports writing directly to destination connectors! Here's what you need to know: Writing to Destinations with PyAirbyte You can use

get_destination()

and the

Destination.write()

method to load data into destinations. You can write either directly from a source or from a cached `ReadResult`:

import airbyte as ab

# Write directly from source to destination
my_source = ab.get_source("source-faker", config={"count": 5_000}, install_if_missing=True)
my_source.select_all_streams()

my_destination = ab.get_destination(
    "destination-foo",
    config={"api_key": "my_api_key"},
    docker_image=True,
)
my_destination.write(my_source)

Or via a read result (useful if you want to validate data in between):

read_result = my_source.read()
# ... optional validation ...
my_destination.write(read_result)

[destinations module] Important Notes on Destinations • Most destination connectors are Java-based, so they require Docker to run. If your environment doesn't support Docker, you may want to use PyAirbyte's built-in SQL caches (DuckDB, Postgres, BigQuery, Snowflake, etc.) instead, which are Python-native and more portable. [destinations module] • SQL caches are recommended over destination connectors when loading to SQL databases, as they don't require Docker. [PyAirbyte reference] State Management PyAirbyte does handle incremental state. The

Destination.write()

method has a

state_cache

parameter for storing incremental state. For incremental syncs, it checks the cache or state cache for existing state values; if none are found, it falls back to destination-specific state, and if neither is available, it performs a full refresh. You can also set

state_cache=False

to disable state management entirely, or use

force_full_refresh=True

to ignore existing state. [destination base] So in summary, PyAirbyte and Airbyte destinations are compatible, not incompatible — PyAirbyte is a valid way to run EL pipelines to destinations without needing the full Airbyte server stack. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai this requires the Airbyte service running though - I don't want to use the airbyte service

Good news — PyAirbyte does not require the Airbyte server/service to run. It is a standalone Python library that runs connectors directly on your machine, without needing

docker-compose up

or the Airbyte web app/server stack. From the docs: > "PyAirbyte isn't an SDK for managing Airbyte. [...] It's a good choice if you're using Jupyter Notebook or iterating on an early prototype for a large data project and don't need to run a server." [platform docs] So with PyAirbyte you can run syncs purely from Python, with no Airbyte server involved. The main caveats to be aware of: • Java-based destination connectors (which is most of them) still require Docker on your machine — but that's just Docker, not the full Airbyte stack. [destinations module] • If you want to avoid Docker entirely, use PyAirbyte's built-in SQL caches (DuckDB, Postgres, Snowflake, BigQuery, MotherDuck) instead of destination connectors. These are Python-native and run without Docker or any server. [PyAirbyte reference] In short: PyAirbyte is specifically designed for the use case you're describing — running data pipelines in Python without the Airbyte service. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai but I have saved data save with the airbyte service - I want to keep using that but instead load in data using pyairbyte

This is a nuanced question — you're asking whether PyAirbyte can pick up incremental state from data previously synced via the Airbyte service, so you don't have to re-sync everything from scratch. Unfortunately, the knowledge sources do not contain enough information to give a confident answer on this specific scenario. What the docs do say about state in PyAirbyte is: > For incremental syncs,

cache

or

state_cache

will be checked for matching state values. If the cache has tracked state, this will be used for the sync. Otherwise, if there is a known destination state, the destination-specific state will be used. If neither are available, a full refresh will be performed. [destination base] This suggests PyAirbyte can use destination-specific state (i.e., state stored at the destination level), but whether it can read state that was previously written and managed by the Airbyte server is not addressed in the available documentation. My honest recommendation would be to: 1. Check the PyAirbyte GitHub repository or Airbyte community forums for more detail on state compatibility between the Airbyte server and PyAirbyte. 2. Consider testing with

force_full_refresh=False

and see if PyAirbyte detects existing destination state — but be prepared that it may fall back to a full refresh if the state formats are incompatible. I wouldn't want to speculate beyond what the sources confirm here. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.