@kapa.ai we have a self hosted Airbyte. Can we setup connection state to be stored in a database?

Something went wrong

@kapa.ai can you try again answering this question?

Thinking completed In Airbyte, connection state is already stored in a database by default [Incremental Data Synchronization]. When running a self-hosted Airbyte instance: 1. Default Storage: Airbyte automatically persists cursor values and connection states into the

state

table inside its internal configuration PostgreSQL database (

airbyte-db

) [Incremental Data Synchronization; Config Database: State]. You do not need to configure extra settings just to have state saved to a database. 2. Using an External Database: If your goal is to store connection states (and other Airbyte metadata) in your own managed external PostgreSQL database (such as AWS RDS or GCP Cloud SQL) instead of the default containerized database, you can configure Airbyte to connect to an external PostgreSQL instance [External Database]: ◦ Docker Compose: Set your database connection details in the

.env

file using variables like

DATABASE_USER

,

DATABASE_PASSWORD

,

DATABASE_HOST

,

DATABASE_PORT

, and

DATABASE_DB

(or the

CONFIG_DATABASE_*

variants) [Forum Post: Error while cloning Airbyte repository]. ◦ Kubernetes (Helm): Disable the built-in database and specify your external instance parameters in

values.yaml

under

global.database

along with an

airbyte-config-secrets

secret [External Database]. Once configured, all connection state information is stored and maintained directly in that PostgreSQL database. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.