soma chandra sekhar attaluri
11/26/2025, 7:09 PMkapa.ai
11/26/2025, 7:09 PMsh docker-compose down docker-compose up -d db 2. Backup the database: sh docker exec airbyte-db pg_dump -U docker airbyte > airbyte_backup.sql 3. Transfer the SQL file to your new (prod) server. 4. On the new server: - Start Airbyte, then stop all services. - Start only the database: sh docker-compose up -d db - Drop and recreate the Airbyte database: sh docker exec airbyte-db psql -U docker -c 'drop database airbyte'; docker exec airbyte-db psql -U docker -c 'create database airbyte with owner docker;' - Restore the backup: sh cat airbyte_backup.sql | docker exec -i airbyte-db psql -U docker -d airbyte - Start all Airbyte services: sh docker-compose up -d - You may need to trigger all connections again to recreate Temporal workflows (can be automated via the Airbyte API) [How to import/export Airbyte to a new instance (docker to docker deploy)](https://discuss.airbyte.io/t/3514) Note: Both Airbyte instances must be on the same version for this to work reliably. --- ### 2. Configuration as Code with Octavia CLI If you want to version control and selectively migrate configurations (sources, destinations, connections), you can use the Octavia CLI: - Export configurations from your source instance: sh octavia init octavia import all This will create YAML files for all resources. - Edit secrets in the YAML files as needed for the new environment. - Apply configurations to your prod instance: sh octavia apply --airbyte-url http://your-prod-airbyte-url --force --airbyte-username USER --airbyte-password PASS [Version control Airbyte configurations with Octavia CLI](https://airbyte.com/tutorials/version-control-airbyte-configurations) Note: Octavia CLI is not officially maintained anymore; consider using the [Airbyte Terraform Provider](https://registry.terraform.io/providers/airbytehq/airbyte/0.1.0) for production use. --- ### 3. Manual Database Dump/Restore (for Kubernetes or External DB) If you are using Kubernetes or an external Postgres database, the process is similar: perform a pg_dump of the Airbyte database and restore it in the new environment. Update your Airbyte deployment to point to the restored database. Important: Airbyte does not automatically migrate or copy data between internal and external databases. You must handle the backup and restore process manually if you change the database backend or move between environments [Connecting to an External Postgres database](https://discuss.airbyte.io/t/2903). --- Summary: - For a full migration, use a Postgres database dump/restore. - For selective or version-controlled migration, use Octavia CLI or Terraform. - Always ensure both Airbyte instances are on the same version to avoid compatibility issues. Let me know if you need step-by-step commands for a specific deployment type!soma chandra sekhar attaluri
11/26/2025, 7:12 PMkapa.ai
11/26/2025, 7:12 PMkapa.ai
11/26/2025, 7:12 PMsh docker exec -it airbyte-db psql -U docker -d airbyte Replace airbyte-db, docker, and airbyte with your actual container name, username, and database name if different. - List tables: sh \dt - Query a table (e.g., list sources): sh SELECT * FROM source; --- ### 2. Move Data to a New Airbyte Database The recommended way is to back up the existing Airbyte database and restore it in the new environment. Here’s how you can do it for a Docker-based deployment: #### A. Backup the Airbyte Database 1. Stop all Airbyte services except the database: sh docker-compose down docker-compose up -d db 2. Create a backup: sh docker exec airbyte-db pg_dump -U docker airbyte > airbyte_backup.sql #### B. Restore the Backup to the New Database 1. Transfer the airbyte_backup.sql file to your new server. 2. On the new server: - Start only the database: sh docker-compose up -d db - Drop and recreate the Airbyte database: sh docker exec airbyte-db psql -U docker -c 'drop database airbyte'; docker exec airbyte-db psql -U docker -c 'create database airbyte with owner docker;' - Restore the backup: sh cat airbyte_backup.sql | docker exec -i airbyte-db psql -U docker -d airbyte - Start all Airbyte services: sh docker-compose up -d Note: Both Airbyte instances must be on the same version for this to work reliably. After restoring, you may need to trigger all connections again to recreate Temporal workflows. This process is described in detail in the [Airbyte community forum](https://discuss.airbyte.io/t/3514). --- If you are using Kubernetes or an external Postgres database, the process is similar: - Use pg_dump to export and psql to import, adjusting connection details as needed. --- Summary: - Use psql to view data in the Airbyte database. - Use pg_dump and psql to move the database to a new Airbyte instance. - Ensure both instances are on the same Airbyte version. For more details and step-by-step instructions, see the [Airbyte forum migration guide](https://discuss.airbyte.io/t/3514).soma chandra sekhar attaluri
11/26/2025, 7:13 PMkapa.ai
11/26/2025, 7:13 PMkapa.ai
11/26/2025, 7:13 PMairbyte-db-0): sh kubectl get pods -n <namespace> | grep db Replace <namespace> with your Airbyte namespace (often airbyte). 2. Connect to the pod: sh kubectl exec -n <namespace> -it airbyte-db-0 -- psql -U airbyte -d db-airbyte - List tables: \dt - Query data: SELECT * FROM source; #### B. If Using an External Database - Connect using psql from your local machine or a bastion host with the credentials you set in your values.yaml and Kubernetes secrets. --- ### 2. Move Data to a New Airbyte Database #### A. Backup the Database 1. Run pg_dump inside the database pod: sh kubectl exec -n <namespace> -it airbyte-db-0 -- bash -c "pg_dump -U airbyte db-airbyte > /tmp/airbyte_backup.sql" 2. Copy the backup to your local machine: sh kubectl cp <namespace>/airbyte-db-0:/tmp/airbyte_backup.sql ./airbyte_backup.sql #### B. Restore the Backup in the New Environment 1. Copy the backup file to the new cluster’s database pod: sh kubectl cp ./airbyte_backup.sql <namespace>/airbyte-db-0:/tmp/airbyte_backup.sql 2. Drop and recreate the schema (this erases all data in the database): sh kubectl exec -n <namespace> -it airbyte-db-0 -- bash -c "psql -U airbyte -d db-airbyte -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'" 3. Restore the database: sh kubectl exec -n <namespace> -it airbyte-db-0 -- bash -c "psql -U airbyte -d db-airbyte < /tmp/airbyte_backup.sql" These steps are directly referenced in the official Airbyte documentation for Helm/Kubernetes deployments and are the recommended approach for moving your Airbyte data between environments installed via Helm and k3s [How to upgrade](https://docs.airbyte.com/platform/1.8/deploying-airbyte/chart-v2-community#how-to-upgrade). --- Note: - If you are using an external Postgres database, perform the pg_dump and psql restore from your local machine or a host with access to the database. - Always ensure both Airbyte instances are running the same version to avoid compatibility issues. - This process will migrate all Airbyte configurations, connections, and state. If you need more specific commands for your setup (e.g., different namespace, external DB), let me know!soma chandra sekhar attaluri
11/26/2025, 7:16 PMkapa.ai
11/26/2025, 7:16 PMkapa.ai
11/26/2025, 7:16 PMairbyte-db-0). You can check for it with: kubectl get pods --all-namespaces | grep db 2. Connect to the database: Use the following command to open a Postgres shell inside the pod: kubectl exec -n <namespace> -it airbyte-db-0 -- psql -U airbyte db-airbyte Replace <namespace> with your Airbyte namespace (often airbyte). 3. List tables: Once inside the Postgres shell, list all tables with: \dt 4. View table contents: To view the contents of a specific table, use a SQL query like: SELECT * FROM <table_name>; These steps are described in the Airbyte documentation for backing up and interacting with the internal Postgres database in a Kubernetes environment [How to upgrade](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#how-to-upgrade). If you are using an external Postgres database, you can connect using any standard Postgres client with the credentials you configured.soma chandra sekhar attaluri
11/26/2025, 7:21 PMkapa.ai
11/26/2025, 7:21 PMkapa.ai
11/26/2025, 7:21 PM\dt in psql) as described in the previous answer, or refer to Airbyte's source code or more detailed database documentation if available.soma chandra sekhar attaluri
11/26/2025, 7:40 PMkapa.ai
11/26/2025, 7:40 PMkapa.ai
11/26/2025, 7:41 PMvalues.yaml In your Helm deployment, add SSO configuration under the global.auth section of your values.yaml file. This enables authentication via your identity provider, allowing multiple users to log in using their organizational credentials. Example configuration snippet: yaml global: auth: instanceAdmin: firstName: "<admin_first_name>" lastName: "<admin_last_name>" emailSecretKey: "<admin_email_secret_key>" passwordSecretKey: "<admin_password_secret_key>" identityProvider: secretName: "<your_k8s_secret_name>" type: "oidc" oidc: domain: "<your_company_domain>" appName: "<your_app_name>" clientIdSecretKey: "<client_id_key>" clientSecretSecretKey: "<client_secret_key>" You will need to set up your identity provider and store the necessary secrets in Kubernetes as described in the [SSO setup guide](https://docs.airbyte.com/platform/access-management/sso#set-up). 3. Apply the configuration and upgrade your Helm release After updating your values.yaml, apply the changes with: helm upgrade <release_name> airbyte/airbyte -n <namespace> -f values.yaml With SSO enabled, all users managed by your identity provider can authenticate to Airbyte, and you can manage their access and roles centrally. This is the recommended and supported way to enable multi-user authentication in Airbyte Enterprise deployed via Helm on k3s [Airbyte Self-Managed Enterprise](https://docs.airbyte.com/platform/1.8/enterprise-setup) [Configure SSO](https://docs.airbyte.com/platform/access-management/sso). > Note: The open-source (Community) edition of Airbyte does not support multi-user authentication or SSO. These features are only available in the Self-Managed Enterprise edition.soma chandra sekhar attaluri
11/26/2025, 7:44 PMkapa.ai
11/26/2025, 7:44 PMkapa.ai
11/26/2025, 7:44 PMsoma chandra sekhar attaluri
11/26/2025, 7:53 PMkapa.ai
11/26/2025, 7:53 PMkapa.ai
11/26/2025, 7:54 PMINSERT, UPDATE, and DELETE operations as they happen, even deletions and intermediate changes. It does not require a cursor column in your tables. - Advantages: - Captures all changes, including deletes. - No need for a dedicated cursor column. - Efficient for large databases (>500GB) and high-change environments. - Minimal impact on source database performance. - When to use: - You need to track deletions. - Your tables lack a suitable cursor column (like updated_at). - Your database is very large or has high change volume. - Limitations: - Requires Postgres v10+ and logical replication enabled. - Only works for tables with primary keys. - Schema changes are not automatically supported; a reset is needed after schema changes. - Additional setup required (replication slot, permissions, etc.) [Cloud SQL for PostgreSQL: Advanced CDC Setup](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#advanced-setup-using-cdc) [CDC Requirements and Limitations](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#connector-limitations) --- ### 2. xmin System Column - How it works: Uses the internal xmin system column, which tracks the transaction ID of the last change to each row. Airbyte scans the table and uses xmin as a cursor to detect new or updated rows. - Advantages: - No need for a user-defined cursor column. - Simpler setup than CDC. - Limitations: - Cannot detect deletions. - Requires a full table scan, which can be slow for large tables. - Row is synced if any column changes, even if not relevant to your sync. - Vulnerable to transaction ID wraparound, which can cause resyncs or missed data. - When to use: - Your database is small (<500GB). - You do not need to track deletions. - You want a simple setup and can tolerate the performance trade-offs. [Xmin Limitations](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#connector-limitations) --- ### 3. User-Defined Column (Standard Incremental) - How it works: You specify a column (commonly updated_at or an auto-incrementing ID) as a cursor. Airbyte queries for rows where the cursor value is greater than the last synced value. - Advantages: - Simple and flexible. - No special database configuration required. - Limitations: - Cannot detect deletions. - Requires a suitable cursor column that reliably tracks changes. - Only captures the latest state at sync time, not intermediate changes. - When to use: - Your tables have a reliable cursor column. - You do not need to track deletions. - Your database is not very large or high-velocity. [Incremental Replication](https://airbyte.com/tutorials/incremental-data-synchronization) --- ### Summary Table | Method | Detects Deletes | Needs Cursor Column | Performance (Large DBs) | Setup Complexity | Use Case | |-----------------------|-----------------|--------------------|-------------------------|------------------|----------------------------------| | CDC (WAL) | Yes | No | Best | High | Large DBs, need deletes tracked | | xmin | No | No | Poor | Medium | Small DBs, simple setup | | User-defined column | No | Yes | Good (if indexed) | Low | Small/medium DBs, simple changes | For more details, see [Postgres Replication Methods](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#postgres-replication-methods) and [CDC vs. Incremental](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication).soma chandra sekhar attaluri
11/26/2025, 8:45 PMkapa.ai
11/26/2025, 8:45 PMkapa.ai
11/26/2025, 8:45 PMsoma chandra sekhar attaluri
11/26/2025, 8:48 PMkapa.ai
11/26/2025, 8:48 PMkapa.ai
11/26/2025, 8:48 PMsoma chandra sekhar attaluri
11/26/2025, 8:52 PMkapa.ai
11/26/2025, 8:52 PMkapa.ai
11/26/2025, 8:53 PMworker.replicaCount value in your values.yaml file for your Helm deployment. For example: yaml worker: replicaCount: 2 # Increase this number as needed This allows more jobs to be processed in parallel, as each worker pod can handle multiple jobs. - Concurrent Sync Limits: You can also control how many syncs can run at once by setting environment variables such as MAX_SYNC_WORKERS and MAX_CHECK_WORKERS for the worker pods. For example: yaml worker: extraEnv: - name: MAX_SYNC_WORKERS value: "5" - name: MAX_CHECK_WORKERS value: "5" Or, in Helm chart V2: yaml worker: maxSyncWorkers: 5 maxCheckWorkers: 5 Adjust these values based on your workload and available resources. - Kubernetes Pod Scheduling: Make sure your Kubernetes cluster has enough resources to schedule the increased number of pods. As a rule of thumb, ensure your cluster can handle up to 2 x <number-of-possible-concurrent-connections> pods at once, since each sync job typically uses two pods (one for reading, one for writing) [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale). Summary: You can scale the number of replication pods by increasing the worker replica count and adjusting the maximum number of concurrent workers. Always monitor your cluster resources to ensure you do not over-provision and cause scheduling issues [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/next/enterprise-setup/scaling-airbyte). If you need to further customize pod resources (CPU, memory), you can also set resource requests and limits in your values.yaml under the jobs.resources section.soma chandra sekhar attaluri
11/27/2025, 3:11 AMkapa.ai
11/27/2025, 3:11 AMkapa.ai
11/27/2025, 3:12 AMworker.replicaCount value in your values.yaml file. This setting controls how many worker pods are deployed, allowing more sync jobs to run in parallel. Here’s how you can do it: yaml worker: replicaCount: <desired_number_of_workers> For example, to run 4 worker pods: yaml worker: replicaCount: 4 Additionally, you can control the number of concurrent syncs each worker can handle by setting environment variables such as MAX_SYNC_WORKERS and MAX_CHECK_WORKERS under `worker.extraEnvs`: yaml worker: extraEnvs: - name: MAX_SYNC_WORKERS value: "5" - name: MAX_CHECK_WORKERS value: "5" By increasing both the number of worker replicas and the maximum number of concurrent workers, you can scale up the number of replication pods and thus the parallelism of your data ingestion. Make sure your Kubernetes cluster has enough resources to handle the increased number of pods. This approach is documented in the Airbyte scaling guides for Helm deployments [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#concurrent-sync-limits).soma chandra sekhar attaluri
11/27/2025, 3:20 AMkapa.ai
11/27/2025, 3:21 AMkapa.ai
11/27/2025, 3:21 AMworker.replicaCount value in your values.yaml file. For example, to set the number of worker pods to 4: yaml worker: replicaCount: 4 After updating your values.yaml, apply the changes with: sh helm upgrade <release_name> airbyte/airbyte -n <namespace> -f values.yaml This will instruct Kubernetes to run the specified number of worker pods, allowing more sync jobs to run in parallel. Adjust this value based on your workload and available cluster resources [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#high-availability). If you need to further control the number of concurrent syncs per worker, you can also set the MAX_SYNC_WORKERS and MAX_CHECK_WORKERS environment variables as described in the documentation.soma chandra sekhar attaluri
11/27/2025, 3:28 AMkapa.ai
11/27/2025, 3:28 AMkapa.ai
11/27/2025, 3:28 AMvalues.yaml file that combines your requested settings for authentication and worker scaling, following the documented structure for Helm chart V1: yaml global: auth: enabled: true cookieSecureSetting: "false" worker: extraEnv: - name: MAX_SYNC_WORKERS value: "5" - name: MAX_CHECK_WORKERS value: "5" - global.auth.enabled: true enables authentication. - global.auth.cookieSecureSetting: "false" disables secure cookies (only use this if you are not using HTTPS, as it reduces security) [Cookie Security Settings](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication#cookie-security-settings). - worker.extraEnv sets the maximum number of concurrent sync and check workers per worker pod [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#concurrent-sync-limits). You can add this to your values.yaml and apply it with your Helm upgrade command. If you want to increase the number of worker pods, you can also add: yaml worker: replicaCount: 2 # or any desired number extraEnv: - name: MAX_SYNC_WORKERS value: "5" - name: MAX_CHECK_WORKERS value: "5" Let me know if you need additional configuration options!