<@U04S8PLJWMT> i want o access airbyte db and use ...
# ask-ai
s
@kapa.ai i want o access airbyte db and use those configurations for another airbyte setup in prod environment how can i do that??
k
To use your Airbyte database configurations from one environment (e.g., dev) in another Airbyte setup (e.g., prod), you have a few options depending on your deployment type and requirements: ### 1. Database Backup and Restore (Recommended for Docker Deployments) You can directly back up the Airbyte Postgres database from your source environment and restore it in your target environment. This method will migrate all configurations, including sources, destinations, connections, and states. Steps: 1. Stop Airbyte services except the database to prevent changes during backup:
Copy code
sh    docker-compose down    docker-compose up -d db
2. Backup the database:
Copy code
sh    docker exec airbyte-db pg_dump -U docker airbyte > airbyte_backup.sql
3. Transfer the SQL file to your new (prod) server. 4. On the new server: - Start Airbyte, then stop all services. - Start only the database:
Copy code
sh      docker-compose up -d db
- Drop and recreate the Airbyte database:
Copy code
sh      docker exec airbyte-db psql -U docker -c 'drop database airbyte';      docker exec airbyte-db psql -U docker -c 'create database airbyte with owner docker;'
- Restore the backup:
Copy code
sh      cat airbyte_backup.sql | docker exec -i airbyte-db psql -U docker -d airbyte
- Start all Airbyte services:
Copy code
sh      docker-compose up -d
- You may need to trigger all connections again to recreate Temporal workflows (can be automated via the Airbyte API) [How to import/export Airbyte to a new instance (docker to docker deploy)](https://discuss.airbyte.io/t/3514) Note: Both Airbyte instances must be on the same version for this to work reliably. --- ### 2. Configuration as Code with Octavia CLI If you want to version control and selectively migrate configurations (sources, destinations, connections), you can use the Octavia CLI: - Export configurations from your source instance:
Copy code
sh   octavia init   octavia import all
This will create YAML files for all resources. - Edit secrets in the YAML files as needed for the new environment. - Apply configurations to your prod instance:
Copy code
sh   octavia apply --airbyte-url http://your-prod-airbyte-url --force --airbyte-username USER --airbyte-password PASS
[Version control Airbyte configurations with Octavia CLI](https://airbyte.com/tutorials/version-control-airbyte-configurations) Note: Octavia CLI is not officially maintained anymore; consider using the [Airbyte Terraform Provider](https://registry.terraform.io/providers/airbytehq/airbyte/0.1.0) for production use. --- ### 3. Manual Database Dump/Restore (for Kubernetes or External DB) If you are using Kubernetes or an external Postgres database, the process is similar: perform a
pg_dump
of the Airbyte database and restore it in the new environment. Update your Airbyte deployment to point to the restored database. Important: Airbyte does not automatically migrate or copy data between internal and external databases. You must handle the backup and restore process manually if you change the database backend or move between environments [Connecting to an External Postgres database](https://discuss.airbyte.io/t/2903). --- Summary: - For a full migration, use a Postgres database dump/restore. - For selective or version-controlled migration, use Octavia CLI or Terraform. - Always ensure both Airbyte instances are on the same version to avoid compatibility issues. Let me know if you need step-by-step commands for a specific deployment type!
s
@kapa.ai How can i do that ?? how can i view the data present in airbyte db and how can i move the data to new airbyte db
k
A message from kapa.ai
To view and move the data present in your Airbyte database to a new Airbyte database, you can follow these steps: --- ### 1. View Data in the Airbyte Database Airbyte stores its configuration (sources, destinations, connections, etc.) in a Postgres database. To view the data: - Access the Airbyte Postgres DB (Docker example):
Copy code
sh   docker exec -it airbyte-db psql -U docker -d airbyte
Replace
airbyte-db
,
docker
, and
airbyte
with your actual container name, username, and database name if different. - List tables:
Copy code
sh   \dt
- Query a table (e.g., list sources):
Copy code
sh   SELECT * FROM source;
--- ### 2. Move Data to a New Airbyte Database The recommended way is to back up the existing Airbyte database and restore it in the new environment. Here’s how you can do it for a Docker-based deployment: #### A. Backup the Airbyte Database 1. Stop all Airbyte services except the database:
Copy code
sh    docker-compose down    docker-compose up -d db
2. Create a backup:
Copy code
sh    docker exec airbyte-db pg_dump -U docker airbyte > airbyte_backup.sql
#### B. Restore the Backup to the New Database 1. Transfer the
airbyte_backup.sql
file to your new server.
2. On the new server: - Start only the database:
Copy code
sh      docker-compose up -d db
- Drop and recreate the Airbyte database:
Copy code
sh      docker exec airbyte-db psql -U docker -c 'drop database airbyte';      docker exec airbyte-db psql -U docker -c 'create database airbyte with owner docker;'
- Restore the backup:
Copy code
sh      cat airbyte_backup.sql | docker exec -i airbyte-db psql -U docker -d airbyte
- Start all Airbyte services:
Copy code
sh      docker-compose up -d
Note: Both Airbyte instances must be on the same version for this to work reliably. After restoring, you may need to trigger all connections again to recreate Temporal workflows. This process is described in detail in the [Airbyte community forum](https://discuss.airbyte.io/t/3514). --- If you are using Kubernetes or an external Postgres database, the process is similar: - Use
pg_dump
to export and
psql
to import, adjusting connection details as needed. --- Summary: - Use
psql
to view data in the Airbyte database. - Use
pg_dump
and
psql
to move the database to a new Airbyte instance. - Ensure both instances are on the same Airbyte version. For more details and step-by-step instructions, see the [Airbyte forum migration guide](https://discuss.airbyte.io/t/3514).
s
@kapa.ai i INstalled it using hel chart and k3s can you be able to help me with the same ?
k
A message from kapa.ai
Certainly! If you installed Airbyte using the Helm chart on k3s, here’s how you can view the data in the Airbyte database and move it to a new Airbyte instance: --- ### 1. View Data in the Airbyte Database (Helm + k3s) Airbyte on Kubernetes (including k3s) typically uses a Postgres database, either internal (inside the cluster) or external (managed, e.g., AWS RDS). #### A. Find the Database Pod (if using internal Postgres) 1. List pods to find the database pod (usually named
airbyte-db-0
):
Copy code
sh    kubectl get pods -n <namespace> | grep db
Replace
<namespace>
with your Airbyte namespace (often
airbyte
). 2. Connect to the pod:
Copy code
sh    kubectl exec -n <namespace> -it airbyte-db-0 -- psql -U airbyte -d db-airbyte
- List tables:
\dt
- Query data:
SELECT * FROM source;
#### B. If Using an External Database - Connect using
psql
from your local machine or a bastion host with the credentials you set in your
values.yaml
and Kubernetes secrets. --- ### 2. Move Data to a New Airbyte Database #### A. Backup the Database 1. Run
pg_dump
inside the database pod:
Copy code
sh    kubectl exec -n <namespace> -it airbyte-db-0 -- bash -c "pg_dump -U airbyte db-airbyte > /tmp/airbyte_backup.sql"
2. Copy the backup to your local machine:
Copy code
sh    kubectl cp <namespace>/airbyte-db-0:/tmp/airbyte_backup.sql ./airbyte_backup.sql
#### B. Restore the Backup in the New Environment 1. Copy the backup file to the new cluster’s database pod:
Copy code
sh    kubectl cp ./airbyte_backup.sql <namespace>/airbyte-db-0:/tmp/airbyte_backup.sql
2. Drop and recreate the schema (this erases all data in the database):
Copy code
sh    kubectl exec -n <namespace> -it airbyte-db-0 -- bash -c "psql -U airbyte -d db-airbyte -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'"
3. Restore the database:
Copy code
sh    kubectl exec -n <namespace> -it airbyte-db-0 -- bash -c "psql -U airbyte -d db-airbyte < /tmp/airbyte_backup.sql"
These steps are directly referenced in the official Airbyte documentation for Helm/Kubernetes deployments and are the recommended approach for moving your Airbyte data between environments installed via Helm and k3s [How to upgrade](https://docs.airbyte.com/platform/1.8/deploying-airbyte/chart-v2-community#how-to-upgrade). --- Note: - If you are using an external Postgres database, perform the
pg_dump
and
psql
restore from your local machine or a host with access to the database. - Always ensure both Airbyte instances are running the same version to avoid compatibility issues. - This process will migrate all Airbyte configurations, connections, and state. If you need more specific commands for your setup (e.g., different namespace, external DB), let me know!
s
@kapa.ai how can i view the contents of the database
k
A message from kapa.ai
To view the contents of the Airbyte database when deployed via Helm and k3s (using the internal Postgres database), you can follow these steps: 1. Open a shell to the Postgres pod: First, find the name of your database pod (commonly
airbyte-db-0
). You can check for it with:
Copy code
kubectl get pods --all-namespaces | grep db
2. Connect to the database: Use the following command to open a Postgres shell inside the pod:
Copy code
kubectl exec -n <namespace> -it airbyte-db-0 -- psql -U airbyte db-airbyte
Replace
<namespace>
with your Airbyte namespace (often
airbyte
). 3. List tables: Once inside the Postgres shell, list all tables with:
Copy code
\dt
4. View table contents: To view the contents of a specific table, use a SQL query like:
Copy code
SELECT * FROM <table_name>;
These steps are described in the Airbyte documentation for backing up and interacting with the internal Postgres database in a Kubernetes environment [How to upgrade](https://docs.airbyte.com/platform/deploying-airbyte/chart-v2-community#how-to-upgrade). If you are using an external Postgres database, you can connect using any standard Postgres client with the credentials you configured.
s
@kapa.ai what are the important tables where my configuration settings are saved
k
A message from kapa.ai
I'm sorry, but the provided knowledge sources do not explicitly list the specific database tables where Airbyte stores configuration settings such as sources, destinations, and connections. The documentation does confirm that Airbyte uses a Postgres database to store all configuration (credentials, frequency, etc.) and job history, but it does not detail the schema or table names involved in this process [Architecture overview](https://docs.airbyte.com/platform/1.8/understanding-airbyte/high-level-view). If you need the exact table names, you may need to inspect the database directly using SQL commands (e.g.,
\dt
in
psql
) as described in the previous answer, or refer to Airbyte's source code or more detailed database documentation if available.
s
@kapa.ai how can we create authentication for more than one user in airbyte installed with helm charts on k3s
k
A message from kapa.ai
To enable authentication for more than one user in Airbyte installed with Helm charts on k3s, you need to use the User Management and Single Sign-On (SSO) features available in Airbyte Self-Managed Enterprise. These features allow multiple users to access and manage Airbyte concurrently, and user access can be managed via your organization's identity provider (such as Okta, Azure Entra ID, or any OIDC-compatible provider). Steps: 1. Deploy Airbyte Self-Managed Enterprise Ensure you are running the Enterprise edition, as multi-user authentication and SSO are only available in this version. 2. Configure SSO in your
values.yaml
In your Helm deployment, add SSO configuration under the
global.auth
section of your
values.yaml
file. This enables authentication via your identity provider, allowing multiple users to log in using their organizational credentials. Example configuration snippet:
Copy code
yaml    global:      auth:        instanceAdmin:          firstName: "<admin_first_name>"          lastName: "<admin_last_name>"          emailSecretKey: "<admin_email_secret_key>"          passwordSecretKey: "<admin_password_secret_key>"        identityProvider:          secretName: "<your_k8s_secret_name>"          type: "oidc"          oidc:            domain: "<your_company_domain>"            appName: "<your_app_name>"            clientIdSecretKey: "<client_id_key>"            clientSecretSecretKey: "<client_secret_key>"
You will need to set up your identity provider and store the necessary secrets in Kubernetes as described in the [SSO setup guide](https://docs.airbyte.com/platform/access-management/sso#set-up). 3. Apply the configuration and upgrade your Helm release After updating your
values.yaml
, apply the changes with:
Copy code
helm upgrade <release_name> airbyte/airbyte -n <namespace> -f values.yaml
With SSO enabled, all users managed by your identity provider can authenticate to Airbyte, and you can manage their access and roles centrally. This is the recommended and supported way to enable multi-user authentication in Airbyte Enterprise deployed via Helm on k3s [Airbyte Self-Managed Enterprise](https://docs.airbyte.com/platform/1.8/enterprise-setup) [Configure SSO](https://docs.airbyte.com/platform/access-management/sso). > Note: The open-source (Community) edition of Airbyte does not support multi-user authentication or SSO. These features are only available in the Self-Managed Enterprise edition.
s
@kapa.ai I am using free open source airbyte version i want to use multi user authentication. can you walk me through how we can do that ?
k
A message from kapa.ai
Airbyte Open Source (the free version) does not support multi-user authentication or user management out of the box. According to the official documentation and community forum, features like user management, role-based access control (RBAC), and Single Sign-On (SSO) are only available in Airbyte Self-Managed Enterprise or Airbyte Cloud editions, not in the open-source version [Securing Airbyte Open Source](https://docs.airbyte.com/platform/1.8/operating-airbyte/security#securing-airbyte-open-source) [Forum: RBAC or permissioning in the UI](https://discuss.airbyte.io/t/97). In the open-source version, authentication is limited to a single admin user configured at setup. There is no built-in way to add or manage multiple users. Workarounds: - You can restrict access to Airbyte by placing it behind a reverse proxy (like NGINX) and using basic authentication or your own SSO/proxy solution to control who can access the Airbyte UI. However, this does not provide true multi-user support within Airbyte itself—everyone will share the same Airbyte session and permissions [Securing Airbyte Open Source](https://docs.airbyte.com/platform/1.8/operating-airbyte/security#securing-airbyte-open-source). - You can also use network-level controls (firewalls, VPNs) to limit who can access the Airbyte instance. Summary: If you require true multi-user authentication and user management, you will need to upgrade to Airbyte Self-Managed Enterprise or Airbyte Cloud. The open-source version does not support this functionality.
s
@kapa.ai can you tell me the differrence between read changes using write ahead log cdc and detect changes with xmin system column and scan chabnges with user defined column ...whats the difference between them and when to use it
k
A message from kapa.ai
Airbyte’s Postgres source connector supports three main methods for detecting and replicating changes: CDC (Change Data Capture) using the Write-Ahead Log (WAL), xmin system column, and user-defined column (standard incremental). Here’s how they differ and when to use each: --- ### 1. CDC (Change Data Capture) using Write-Ahead Log (WAL) - How it works: Airbyte reads changes directly from the Postgres WAL using logical replication. This method captures all
INSERT
,
UPDATE
, and
DELETE
operations as they happen, even deletions and intermediate changes. It does not require a cursor column in your tables. - Advantages: - Captures all changes, including deletes. - No need for a dedicated cursor column. - Efficient for large databases (>500GB) and high-change environments. - Minimal impact on source database performance. - When to use: - You need to track deletions. - Your tables lack a suitable cursor column (like
updated_at
). - Your database is very large or has high change volume. - Limitations: - Requires Postgres v10+ and logical replication enabled. - Only works for tables with primary keys. - Schema changes are not automatically supported; a reset is needed after schema changes. - Additional setup required (replication slot, permissions, etc.) [Cloud SQL for PostgreSQL: Advanced CDC Setup](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#advanced-setup-using-cdc) [CDC Requirements and Limitations](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#connector-limitations) --- ### 2. xmin System Column - How it works: Uses the internal
xmin
system column, which tracks the transaction ID of the last change to each row. Airbyte scans the table and uses
xmin
as a cursor to detect new or updated rows. - Advantages: - No need for a user-defined cursor column. - Simpler setup than CDC. - Limitations: - Cannot detect deletions. - Requires a full table scan, which can be slow for large tables. - Row is synced if any column changes, even if not relevant to your sync. - Vulnerable to transaction ID wraparound, which can cause resyncs or missed data. - When to use: - Your database is small (<500GB). - You do not need to track deletions. - You want a simple setup and can tolerate the performance trade-offs. [Xmin Limitations](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#connector-limitations) --- ### 3. User-Defined Column (Standard Incremental) - How it works: You specify a column (commonly
updated_at
or an auto-incrementing ID) as a cursor. Airbyte queries for rows where the cursor value is greater than the last synced value. - Advantages: - Simple and flexible. - No special database configuration required. - Limitations: - Cannot detect deletions. - Requires a suitable cursor column that reliably tracks changes. - Only captures the latest state at sync time, not intermediate changes. - When to use: - Your tables have a reliable cursor column. - You do not need to track deletions. - Your database is not very large or high-velocity. [Incremental Replication](https://airbyte.com/tutorials/incremental-data-synchronization) --- ### Summary Table | Method | Detects Deletes | Needs Cursor Column | Performance (Large DBs) | Setup Complexity | Use Case | |-----------------------|-----------------|--------------------|-------------------------|------------------|----------------------------------| | CDC (WAL) | Yes | No | Best | High | Large DBs, need deletes tracked | | xmin | No | No | Poor | Medium | Small DBs, simple setup | | User-defined column | No | Yes | Good (if indexed) | Low | Small/medium DBs, simple changes | For more details, see [Postgres Replication Methods](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#postgres-replication-methods) and [CDC vs. Incremental](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication).
s
@kapa.ai Is there any limit for sync number of records or sync size of data in airbyte open source version
k
A message from kapa.ai
In Airbyte Open Source, there are no hard-coded limits on the number of records or the total sync size imposed by the Airbyte platform itself. You can sync as much data as your infrastructure (CPU, memory, disk, and database) can handle. This is explicitly stated in the license FAQ: "no limitations on volume, number of users, number of connections…" for open-source users [License FAQ](https://docs.airbyte.com/community/licenses/license-faq#faq). However, practical limits may arise from: - Resource constraints (memory, CPU, disk) on your deployment. - Connector-specific or destination-specific limits (e.g., Google Sheets has a 10 million cell limit per document) [Google Sheets Limitations](https://docs.airbyte.com/integrations/destinations/google-sheets#limitations). - Batch sizes and memory settings (e.g., destinations process records in batches, and large row sizes can cause out-of-memory errors) [Forum: Memory request and limit](https://discuss.airbyte.io/t/2214). Summary: Airbyte Open Source does not impose a platform-level limit on the number of records or total sync size, but you may encounter limits based on your infrastructure or the specific connectors/destinations you use.
s
@kapa.ai what do you mean by infrastructure ....in my sync process i am doing a sync for 132 tables after going after 50 million records its starting from the tables which it already synced in ..what might be the issue
k
A message from kapa.ai
In the context of Airbyte, "infrastructure" refers to the computing resources that your Airbyte deployment uses—such as CPU, memory (RAM), disk space, and network capacity. These resources are provided by the servers, virtual machines, or Kubernetes nodes where Airbyte and its database are running. Regarding your issue: If your sync process is restarting from tables that were already synced after reaching around 50 million records, this could be due to resource exhaustion (such as running out of memory or CPU), or a failure in the sync process that causes Airbyte to retry the job. Airbyte is designed as an "at-least-once" delivery system, meaning that if a sync fails partway through, it may reprocess and resend records from the last successful checkpoint or chunk. This can result in previously synced tables being processed again if the sync is not fully completed before the failure or interruption [Resumability & Resumable Full Refresh](https://docs.airbyte.com/platform/1.8/understanding-airbyte/resumability). Common causes for this behavior include: - Out-of-memory (OOM) errors: If the sync process consumes more memory than available, the job may be killed and restarted, causing partial progress to be lost and tables to be re-synced from the last checkpoint [Forum: CockroachDB Source Connector Failures at >10M records](https://discuss.airbyte.io/t/1261), [Forum: Source Postgres: source cannot be stopped!](https://discuss.airbyte.io/t/2109). - Job failures or timeouts: Any error during the sync can cause Airbyte to retry, leading to duplicate processing if the sync mode is not deduplicating. - Sync mode: If you are using "Full Refresh Append," Airbyte will append data on each retry, potentially causing duplicates [Forum: Airbyte committing records for failed sync](https://discuss.airbyte.io/t/1064). What you can do: - Monitor your Airbyte deployment's resource usage (CPU, memory) and increase resources if you see high utilization. - Check Airbyte and connector logs for OOM or other errors. - Consider using "Full Refresh Overwrite" or an incremental sync mode with deduplication if you want to avoid duplicate records on retries. If you need more specific troubleshooting, reviewing the logs for errors around the time the sync restarts would be helpful.
s
@kapa.ai is there any way we can customize the replication pods inside the airbyte installed on kubernetes what if i want to increase the number of pods replicating
k
A message from kapa.ai
Yes, you can customize and scale the replication pods (i.e., the pods that run sync jobs) in Airbyte when installed on Kubernetes. How to increase the number of pods: - Worker Replicas: You can increase the number of Airbyte worker pods by setting the
worker.replicaCount
value in your
values.yaml
file for your Helm deployment. For example:
Copy code
yaml   worker:     replicaCount: 2  # Increase this number as needed
This allows more jobs to be processed in parallel, as each worker pod can handle multiple jobs. - Concurrent Sync Limits: You can also control how many syncs can run at once by setting environment variables such as
MAX_SYNC_WORKERS
and
MAX_CHECK_WORKERS
for the worker pods. For example:
Copy code
yaml   worker:     extraEnv:       - name: MAX_SYNC_WORKERS         value: "5"       - name: MAX_CHECK_WORKERS         value: "5"
Or, in Helm chart V2:
Copy code
yaml   worker:     maxSyncWorkers: 5     maxCheckWorkers: 5
Adjust these values based on your workload and available resources. - Kubernetes Pod Scheduling: Make sure your Kubernetes cluster has enough resources to schedule the increased number of pods. As a rule of thumb, ensure your cluster can handle up to
2 x <number-of-possible-concurrent-connections>
pods at once, since each sync job typically uses two pods (one for reading, one for writing) [Scaling Airbyte](https://docs.airbyte.com/platform/operator-guides/scaling-airbyte#what-to-scale). Summary: You can scale the number of replication pods by increasing the worker replica count and adjusting the maximum number of concurrent workers. Always monitor your cluster resources to ensure you do not over-provision and cause scheduling issues [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/next/enterprise-setup/scaling-airbyte). If you need to further customize pod resources (CPU, memory), you can also set resource requests and limits in your
values.yaml
under the
jobs.resources
section.
s
@kapa.ai how to increase number of replication pods required to ingest data through values.yaml file in helm v1
k
A message from kapa.ai
To increase the number of replication pods (i.e., the pods responsible for running sync jobs) in Airbyte when using Helm chart V1, you should adjust the
worker.replicaCount
value in your
values.yaml
file. This setting controls how many worker pods are deployed, allowing more sync jobs to run in parallel. Here’s how you can do it:
Copy code
yaml worker:   replicaCount: <desired_number_of_workers>
For example, to run 4 worker pods:
Copy code
yaml worker:   replicaCount: 4
Additionally, you can control the number of concurrent syncs each worker can handle by setting environment variables such as
MAX_SYNC_WORKERS
and
MAX_CHECK_WORKERS
under `worker.extraEnvs`:
Copy code
yaml worker:   extraEnvs:     - name: MAX_SYNC_WORKERS       value: "5"     - name: MAX_CHECK_WORKERS       value: "5"
By increasing both the number of worker replicas and the maximum number of concurrent workers, you can scale up the number of replication pods and thus the parallelism of your data ingestion. Make sure your Kubernetes cluster has enough resources to handle the increased number of pods. This approach is documented in the Airbyte scaling guides for Helm deployments [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#concurrent-sync-limits).
s
@kapa.ai How to replicate the pods to the desired number
k
A message from kapa.ai
To replicate (scale) the Airbyte worker pods to your desired number when using Helm (chart V1), you need to set the
worker.replicaCount
value in your
values.yaml
file. For example, to set the number of worker pods to 4:
Copy code
yaml worker:   replicaCount: 4
After updating your
values.yaml
, apply the changes with:
Copy code
sh helm upgrade <release_name> airbyte/airbyte -n <namespace> -f values.yaml
This will instruct Kubernetes to run the specified number of worker pods, allowing more sync jobs to run in parallel. Adjust this value based on your workload and available cluster resources [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#high-availability). If you need to further control the number of concurrent syncs per worker, you can also set the
MAX_SYNC_WORKERS
and
MAX_CHECK_WORKERS
environment variables as described in the documentation.
s
@kapa.ai global: auth: enabled: true cookieSecureSetting: "false" worker: extraEnv: ## We recommend setting both environment variables with a single, shared value. - name: MAX_SYNC_WORKERS value: ## e.g. 5 - name: MAX_CHECK_WORKERS value: ## e.g. 5 give me everything together in values.yaml file
k
A message from kapa.ai
Certainly! Here is an example of a
values.yaml
file that combines your requested settings for authentication and worker scaling, following the documented structure for Helm chart V1:
Copy code
yaml global:   auth:     enabled: true     cookieSecureSetting: "false"  worker:   extraEnv:     - name: MAX_SYNC_WORKERS       value: "5"     - name: MAX_CHECK_WORKERS       value: "5"
-
global.auth.enabled: true
enables authentication. -
global.auth.cookieSecureSetting: "false"
disables secure cookies (only use this if you are not using HTTPS, as it reduces security) [Cookie Security Settings](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication#cookie-security-settings). -
worker.extraEnv
sets the maximum number of concurrent sync and check workers per worker pod [Scaling Airbyte After Installation](https://docs.airbyte.com/platform/1.8/enterprise-setup/scaling-airbyte#concurrent-sync-limits). You can add this to your
values.yaml
and apply it with your Helm upgrade command. If you want to increase the number of worker pods, you can also add:
Copy code
yaml worker:   replicaCount: 2  # or any desired number   extraEnv:     - name: MAX_SYNC_WORKERS       value: "5"     - name: MAX_CHECK_WORKERS       value: "5"
Let me know if you need additional configuration options!