soma chandra sekhar attaluri
11/25/2025, 7:03 PMkapa.ai
11/25/2025, 7:03 PMabctl), but this is not recommended for high-scale, production workloads ([see here](https://docs.airbyte.com/platform/deploying-airbyte/abctl-ec2)). ### 2. Helm Charts on Kubernetes (EKS/k3s) vs. abctl - Helm Charts on Kubernetes (Recommended): - Flexibility: Helm is the recommended way to deploy Airbyte on Kubernetes, allowing you to customize your deployment for your cloud provider and integrations (e.g., external databases, loggers, ingresses). - Production-Readiness: The documentation highly recommends Helm for production deployments, especially on EKS ([see here](https://docs.airbyte.com/platform/deploying-airbyte)). - k3s: While k3s is a lightweight Kubernetes distribution and can be used, for your scale and production needs, EKS is preferred over k3s. - abctl (for Local/Small Deployments): - Use Case: abctl is designed for quick, local, or small-scale deployments, not for large-scale, production workloads ([see here](https://docs.airbyte.com/platform/deploying-airbyte/abctl-ec2)). ### 3. Underlying OS: Linux vs. Windows - Linux (Recommended): - Compatibility: All Airbyte deployment guides and scripts assume a Linux environment (Amazon Linux, Ubuntu, etc.). - Docker Support: Linux is the standard for Docker and Kubernetes, and all Airbyte installation steps are documented for Linux. - Windows: Not recommended or documented for Airbyte server deployments ([see here](https://docs.airbyte.com/platform/deploying-airbyte/abctl-ec2)). --- ## Summary Recommendation - Deploy Airbyte on AWS EKS (Kubernetes) using Helm charts, with Linux-based, memory-optimized EC2 worker nodes. - This approach gives you the scalability, reliability, and maintainability needed for high-volume, production-grade data ingestion. - Avoid single-node EC2 or Windows-based deployments for your use case. Why? EKS with Helm on Linux is the best-practice, production-ready approach recommended by Airbyte for large-scale, resilient, and manageable deployments on AWS. It allows you to scale horizontally, recover from failures, and integrate with AWS services (S3, RDS, Secrets Manager) efficiently. Single EC2 or Windows deployments are not recommended for your scale or for production environments. References: - [Enterprise Implementation Guide: Infrastructure Prerequisites](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#infrastructure-prerequisites) - [Deploying Airbyte: Understanding Deployment](https://docs.airbyte.com/platform/deploying-airbyte) - [Using an EC2 Instance with abctl](https://docs.airbyte.com/platform/deploying-airbyte/abctl-ec2)soma chandra sekhar attaluri
11/25/2025, 8:15 PMkapa.ai
11/25/2025, 8:15 PMkapa.ai
11/25/2025, 8:15 PMvalues.yaml file. The documentation provides clear examples for configuring authentication, including setting admin credentials and integrating with identity providers [Step 2: Configure your Deployment](https://docs.airbyte.com/platform/enterprise-setup/implementation-guide#step-2-configure-your-deployment). For example, to enable authentication and set admin credentials in your `values.yaml`: yaml global: auth: instanceAdmin: firstName: "YourFirstName" lastName: "YourLastName" emailSecretKey: "admin-email" passwordSecretKey: "admin-password" You can also disable authentication if needed: yaml global: auth: enabled: false Summary: - Helm chart installation is supported for Airbyte 2.0.1. - Authentication (including SSO for enterprise) is fully supported and configurable via Helm charts. If you need more specific configuration examples or have a particular authentication method in mind, let me know!soma chandra sekhar attaluri
11/25/2025, 8:16 PMkapa.ai
11/25/2025, 8:16 PMkapa.ai
11/25/2025, 8:16 PMvalues.yaml file and apply them to your running deployment. Steps: 1. Edit your values.yaml file Add or update the following section to enable authentication and set admin credentials: yaml global: auth: instanceAdmin: firstName: "YourFirstName" lastName: "YourLastName" # Optionally, configure email and password via Kubernetes secrets # emailSecretKey: "admin-email" # passwordSecretKey: "admin-password" If you want to disable authentication, you can set: yaml global: auth: enabled: false 2. Apply the updated configuration Upgrade your Helm release with the updated `values.yaml`: sh helm upgrade \ --namespace airbyte \ --values ./values.yaml \ [RELEASE_NAME] \ --version 1.9.2 \ airbyte/airbyte 3. (Optional) Set admin password via Kubernetes Secret If you want to control the admin password, create a Kubernetes secret: yaml apiVersion: v1 kind: Secret metadata: name: airbyte-auth-secrets type: Opaque stringData: instance-admin-password: yourpassword Apply it with: sh kubectl apply -f secret.yaml -n <YOUR_NAMESPACE> You may need to restart the airbyte-server pod for changes to take effect. Note: - The documentation for Airbyte 1.8 and 1.7 shows that authentication is managed via the values.yaml file and Kubernetes secrets, and this approach is consistent across minor versions, including 1.9.2 ([Authentication - 1.8](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication), [Authentication - 1.7](https://docs.airbyte.com/platform/1.7/deploying-airbyte/integrations/authentication)). - If you need SSO or OIDC integration, you can add the relevant configuration under the auth section as described in the implementation guides. If you need more specific examples for SSO or OIDC, let me know!soma chandra sekhar attaluri
11/25/2025, 8:18 PMkapa.ai
11/25/2025, 8:18 PMkapa.ai
11/25/2025, 8:18 PMairbyte-auth-secrets or airbyte-config-secrets, depending on your configuration. These secrets are referenced in your values.yaml file to bootstrap authentication for the Airbyte instance. You can view these credentials by running: sh kubectl get secret airbyte-auth-secrets -n <YOUR_NAMESPACE> -o yaml The secret will contain keys like instance-admin-password, instance-admin-client-id, and instance-admin-client-secret, which are base64 encoded. You can decode them using the base64 -d command [Authentication (Next)](https://docs.airbyte.com/platform/next/deploying-airbyte/integrations/authentication). If you want to set or update these credentials, you can create or modify the Kubernetes Secret and apply it to your cluster. For example: yaml apiVersion: v1 kind: Secret metadata: name: airbyte-auth-secrets type: Opaque stringData: instance-admin-password: yourpassword Apply it with: sh kubectl apply -f secret.yaml -n <YOUR_NAMESPACE> This approach is consistent across Airbyte versions and is the recommended way to securely store authentication credentials in a Kubernetes deployment [Authentication (1.8)](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication).soma chandra sekhar attaluri
11/25/2025, 8:26 PMkapa.ai
11/25/2025, 8:26 PMkapa.ai
11/25/2025, 8:26 PMabctl and with Helm charts is due to how authentication is configured by default in each deployment method. - abctl Installations: When you install Airbyte using abctl, authentication is enabled by default. You are required to log in with a user ID (email) and password. The credentials are generated and can be retrieved using abctl local credentials. This ensures that access to the Airbyte UI is protected out of the box [Authentication (1.8)](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication). - Helm Chart Installations: When deploying Airbyte with Helm charts, authentication is configurable via the values.yaml file. If you do not explicitly enable authentication in your values.yaml, Airbyte may run without requiring a user ID and password. This is why you might not see a login prompt after a default Helm installation [Authentication (1.8)](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication). How to resolve this: To enable authentication in your Helm-based Airbyte deployment, update your values.yaml file to include: yaml global: auth: enabled: true You can also specify admin credentials using Kubernetes secrets as described in the documentation. After updating your values.yaml, upgrade your Helm release: sh helm upgrade \ --namespace <YOUR_NAMESPACE> \ --values ./values.yaml \ <RELEASE_NAME> \ airbyte/airbyte If you want to set a specific admin password, create a Kubernetes secret named airbyte-auth-secrets with the appropriate keys (e.g., instance-admin-password) and apply it to your namespace. You may need to restart the Airbyte server pod for the changes to take effect [Authentication (1.8)](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication). Summary: - abctl enables authentication by default. - Helm chart deployments require you to explicitly enable authentication in values.yaml. - To require login on Helm deployments, set global.auth.enabled: true and configure credentials as needed. If you need more details on setting up admin credentials or SSO, let me know!soma chandra sekhar attaluri
11/25/2025, 8:38 PMkapa.ai
11/25/2025, 8:38 PMkapa.ai
11/25/2025, 8:38 PMabctl to install Airbyte: Add the --insecure-cookies flag to your install command. For example: sh abctl local install --host example.com --insecure-cookies This tells Airbyte not to require secure cookies, which are only supported over HTTPS. This is the recommended fix for this error when deploying with abctl ([see documentation](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#make-sure-you-have-disabled-secure-cookies)). #### If you are using Helm charts: Update your values.yaml file to include: yaml global: auth: cookieSecureSetting: "false" Or, for Helm chart V2: yaml global: auth: security: cookieSecureSetting: "false" Then upgrade your Helm release with the updated values.yaml. This will disable secure cookies for your deployment ([see documentation](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication#cookie-security-settings)). --- Summary: - Use --insecure-cookies with abctl, or - Set cookieSecureSetting: "false" in your values.yaml for Helm deployments. This will resolve the cookie error when running Airbyte over HTTP.soma chandra sekhar attaluri
11/25/2025, 8:43 PMkapa.ai
11/25/2025, 8:43 PMkapa.ai
11/25/2025, 8:43 PMvalues.yaml snippets for both enabling authentication and disabling secure cookies in Airbyte, as well as for disabling authentication if needed. --- ### Enable Authentication & Disable Secure Cookies For Helm chart V1: yaml global: auth: enabled: true cookieSecureSetting: "false" For Helm chart V2: yaml global: auth: enabled: true security: cookieSecureSetting: "false" This configuration enables authentication and ensures cookies are not required to be secure, which is necessary when running Airbyte over HTTP ([source](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication#cookie-security-settings)). --- ### Disable Authentication For Helm chart V1: yaml global: auth: enabled: false For Helm chart V2: yaml global: auth: enabled: false This will turn off authentication entirely ([source](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication)). --- Note: - Use the security: subkey for Helm chart V2. - After updating your values.yaml, apply the changes with a Helm upgrade. If you need to adjust the SameSite cookie setting as well, you can add: yaml cookieSameSiteSetting: "None" or yaml security: cookieSameSiteSetting: "None" as appropriate for your chart version ([source](https://docs.airbyte.com/platform/1.8/deploying-airbyte/integrations/authentication#cookie-security-settings)).soma chandra sekhar attaluri
11/26/2025, 12:29 AMsoma chandra sekhar attaluri
11/26/2025, 12:30 AMkapa.ai
11/26/2025, 12:30 AMkapa.ai
11/26/2025, 12:30 AM, or \t) and quote characters (default is "), and does not provide explicit guidance on using non-printable ASCII characters such as 0x1F (unit separator) as the field delimiter or 0x1E (record separator) as the quote character. Here are the relevant configuration options for CSV in Airbyte: - Delimiter: You can specify a custom delimiter character. - Quote Character: You can specify a custom quote character. - Record Delimiter: There is no explicit mention of configuring a custom record delimiter; the default is \n. - Escape Character: You can specify an escape character. Example from the documentation: > The delimiter field lets you specify which character acts as the separator. By default, this value is set to ,. > Quote Character: By default, this is set to " [CSV File Format Settings](https://docs.airbyte.com/integrations/sources/s3#file-format-settings). However, the documentation does not specify whether you can use hexadecimal or non-printable ASCII characters (like 0x1F or 0x1E) directly in the configuration. It also does not mention a way to change the record delimiter from \n to another character. Summary: - You can customize the delimiter and quote character for CSV output in Airbyte, but the documentation does not confirm support for non-printable ASCII characters (such as 0x1F or 0x1E) or changing the record delimiter. - There is no explicit example or guarantee in the knowledge sources that your specific requirements (field delimiter 0x1F, record delimiter \n, quote character 0x1E) are supported. If you need this level of customization, you may need to test it directly or consider post-processing the output. If you have further questions or need more advanced customization, you may want to reach out to the Airbyte community or support channels. --- References: - [CSV File Format Settings (S3 Source)](https://docs.airbyte.com/integrations/sources/s3#file-format-settings) - [CSV File Format Settings (SharePoint Enterprise)](https://docs.airbyte.com/integrations/enterprise-connectors/source-sharepoint-enterprise#file-format-settings)soma chandra sekhar attaluri
11/26/2025, 12:37 AMkapa.ai
11/26/2025, 12:37 AMkapa.ai
11/26/2025, 12:37 AM0x1F for the field delimiter and 0x1E for the quote character) in the CSV output format. The documentation confirms that you can customize the delimiter and quote character for CSV files, but only common characters like commas, tabs, or double quotes are mentioned. There is no mention of supporting hexadecimal or non-printable ASCII values for these settings, nor is there an option to change the record delimiter from the default \n [CSV File Format Settings](https://docs.airbyte.com/integrations/sources/s3#file-format-settings). Therefore, based on the available documentation, it is unclear whether Airbyte supports this level of customization for CSV output. If this functionality is critical, you may need to consider post-processing the output or consult the Airbyte community for further guidance.soma chandra sekhar attaluri
11/26/2025, 12:38 AMkapa.ai
11/26/2025, 12:38 AMkapa.ai
11/26/2025, 12:39 AM\t), or other printable characters, and you can choose the quote character (default is "). However, the documentation only provides examples and guidance for common, printable characters (like commas, tabs, or double quotes) and does not mention support for non-printable ASCII characters (such as 0x1F for the field delimiter or 0x1E for the quote character) in the Airbyte UI. There is also no mention of customizing the record delimiter (which defaults to \n) in the UI. So, while you can set the delimiter and quote character in the Airbyte UI, the knowledge sources do not confirm that you can use non-printable ASCII values for these settings. If you need to use such characters, you may need to test directly in the UI or consider post-processing the output outside of Airbyte [CSV File Format Settings](https://docs.airbyte.com/integrations/sources/s3#file-format-settings).