@kapa.ai can you talk me through data security with Airbyte? What protocols are in place to ensure the secure transfer of sensetive data?

Airbyte follows industry-standard security practices and provides multiple layers of protection to ensure the secure transfer and handling of sensitive data [Securing Airbyte]. 1. Encryption in Transit and at Rest • Data in Transit: ◦ Airbyte connectors pull and transfer data across encrypted channels, utilizing TLS, SSL, HTTPS, or SSH tunneling [Securing Airbyte Cloud; Securing Airbyte Open Source]. ◦ Individual database connectors (e.g., Oracle) support dedicated encryption options such as TLS with certificate verification and Native Network Encryption (NNE) with algorithms like AES-256 [Oracle Encryption options]. • Data at Rest & Ephemeral Processing: ◦ Airbyte operates as a data pipeline that moves data from source to destination. As soon as data is transferred, it is purged from the deployment, meaning sync payload data is not persistently stored [Securing Airbyte; Securing Airbyte Cloud]. ◦ Any retained customer metadata (e.g., table/column names) is stored encrypted using cloud-native encryption services with AES-256 keys [Securing Airbyte; Securing Airbyte Cloud]. 2. Network Isolation and Private Connectivity • Pod-Level Isolation: All sync jobs run in separate, dedicated Kubernetes pods to ensure complete workload and data isolation between syncs and organizations [Private Links How it works]. • Private Links (AWS PrivateLink): Airbyte Cloud supports Private Links, enabling data transfer via dedicated VPC endpoints rather than over the public internet. This prevents resources from requiring public IP addresses or open firewall rules [Private Links How it works]. • Self-Managed Security: For Airbyte Open Source, deployments can be secured within private networks/VPCs, placed behind reverse proxies (for SSL termination), or protected via tools like GCP Identity-Aware Proxy or AWS Session Manager [Securing Airbyte Open Source]. 3. Credential and Secret Management • Airbyte Cloud: Credentials (API keys, passwords) are transmitted over HTTPS and stored securely in Google Cloud Secret Manager. Configurations stored in databases or disks only hold references/pointers to the secret rather than the raw secret itself [Securing Airbyte Cloud]. • Airbyte Open Source: Secrets are stored in the internal application database by default, or users can configure Google Secret Manager for external secret management [Securing Airbyte Open Source]. • Token Lifecycles: Platform tokens are short-lived (e.g., application tokens expire after 15 minutes), and stored credentials are never returned in API responses after creation [Connectors and credentials Security]. 4. In-Flight Data Protection (Field Mappings) Airbyte offers built-in field-level mapping features to protect sensitive or personally identifiable information (PII) before loading it into your destination [Mappings]: • Hashing: Irreversible masking using algorithms such as SHA-256, SHA-512, and MD5 [Mappings]. • Encryption: Reversible protection using RSA encryption with user-provided keys [Mappings]. 5. Access Control and Compliance • Access Control: Airbyte Cloud and Enterprise provide Role-Based Access Control (RBAC) to enforce least-privilege user access [Securing Airbyte Cloud]. • Compliance: Airbyte Cloud undergoes regular third-party audits and maintains SOC 2 Type II compliance and ISO 27001 certification [Securing Airbyte Cloud]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

Please refine the above to only information relevant to airbyte open source @kapa.ai

Here is the data security information specifically relevant to Airbyte Open Source: 1. Ephemeral Data Processing and Encryption • Data Purging: Airbyte acts as a data pipeline. As soon as data is extracted from the source and transferred to the destination, it is purged from the deployment [Securing Airbyte]. • Encryption in Transit: Most Airbyte Open Source connectors support encryption-in-transit via SSL, HTTPS, or SSH tunnels. It is recommended to configure connectors to use encryption whenever available [Securing Airbyte Open Source]. 2. Network Security and Host Protection Because Open Source security depends on your infrastructure setup, controlling network and hardware access is your responsibility [Securing Airbyte Open Source]: • Private Networks and Firewalls: Deploy Airbyte within a private network or use firewalls to filter IP addresses allowed to access the host [Securing Airbyte Open Source]. • Reverse Proxies: Deploy Airbyte behind a reverse proxy (e.g., NGINX) to manage access control and handle SSL/TLS termination [Securing Airbyte Open Source]. • Cloud Access Proxies: If hosting on a cloud provider, secure access using services like GCP Identity-Aware Proxy (IAP) or AWS Systems Manager Session Manager [Securing Airbyte Open Source]. • Security Self-Checks: In version 0.44.0 and later, Airbyte Open Source runs a setup self-check to verify if the instance is exposed to the internet and if strong authentication is configured [Securing Airbyte Open Source]. 3. Credential and Secret Management • Database Storage: By default, Airbyte stores credentials (such as API keys and passwords) in the internal Airbyte application database [Securing Airbyte Open Source]. • External Secret Manager: If deploying on GCP, you can configure Airbyte Open Source to store credentials in Google Secret Manager instead of the database by setting the

SECRET_PERSISTENCE

environment variable along with your GCP credentials [Securing Airbyte Open Source]. 4. Instance Authentication and Access Controls • Built-in Authentication: Airbyte generates a secure default password upon deployment (via

abctl

or Helm), which can be managed or replaced using Kubernetes secrets [Authentication]. • RBAC Limitations: Airbyte Open Source does not include built-in user management or role-based access control (RBAC) to restrict access within the UI or API [Securing Airbyte Open Source]. 5. Field-Level Data Obscuration (Mappings) To protect sensitive data before it reaches the destination, you can configure stream field mappings directly [Mappings]: • Hashing: Irreversibly obscure fields using algorithms like MD5, SHA-256, and SHA-512 [Mappings]. • Encryption: Reversibly protect fields using RSA encryption with hex-encoded DER public keys [Mappings]. 6. Technical and Sensitive Log Handling • Initial user configurations may be output to the log stream if the server log level is set to

DEBUG

. It is recommended to keep the log level set to

INFO

during normal operation to keep configuration details private [Securing Airbyte Open Source]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai if there is no encryption option for a connector, do i just assume the data is not encrypted? for example, docs.airbyte.com/integrations/sources/google-analytics-data-api ?

No, you should not assume the data is unencrypted. For API-based connectors like Google Analytics (GA4) Data API, encryption in transit is enforced automatically at the protocol level: • Mandatory HTTPS: Airbyte connector standards require all connectors to use HTTPS only when communicating with external services [Airbyte connectors QA checks]. • Underlying API Endpoints: The Google Analytics Data API connector communicates directly with Google's secure HTTPS endpoints (such as

https://analyticsdata.googleapis.com

and

https://oauth2.googleapis.com

) using TLS encryption for all authentication and data retrieval requests [Source-google-analytics-data-api manifest; Source-google-analytics-data-api metadata]. Explicit encryption toggle settings (like SSL/TLS modes or SSH tunnels) are generally only present on database connectors (e.g., PostgreSQL, MySQL), where databases can be configured to accept either encrypted or unencrypted connections. For REST API sources like Google Analytics, communication is always encrypted in transit over HTTPS [Airbyte connectors QA checks; Securing Airbyte Open Source]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai is the airbyte connectors QA checks applicable to marketplace connectors, or just airbyte published ones?

This message contains interactive elements.

@kapa.ai is the airbyte connectors QA checks applicable to marketplace connectors, or just airbyte published ones?

The provided knowledge sources do not explicitly mention "marketplace connectors" or specify whether the QA checks apply to third-party/marketplace connectors versus only Airbyte-published connectors. According to the documentation, these static-analysis checks run in Airbyte's CI/CD pipeline for connectors in the Airbyte codebase to ensure they follow best practices, meet Airbyte standards, and can be safely integrated into the Airbyte platform and released to registries (like DockerHub, PyPI, etc.) [Airbyte connectors QA checks]. Most individual checks state they apply to connectors with "any support level" and "any internal support level" (with some specific checks targeted at certified support levels) [Documentation; Metadata]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai cane one still create custom connectors on airbyte cloud?

Yes, you can still create custom connectors on Airbyte Cloud [Connector Development; Airbyte 1.6 Release Notes]. However, there are specific limitations and methods for doing so on Airbyte Cloud: • Connector Builder (No-Code / Low-Code): The primary and recommended method for Airbyte Cloud is using the Connector Builder [Connector Development]. You can start from scratch, import a low-code YAML manifest, or load an existing connector [How to add custom source to PyAirbyte]. Once built and tested, you can publish it directly to your workspace [Connector Development]. • API and Terraform: You can programmatically create and manage custom declarative (YAML) source definitions on Airbyte Cloud using the API and Terraform provider [Airbyte 1.6 Release Notes]. • Docker-based Custom Connectors: On the Cloud version of Airbyte, you can generally only create or modify custom connectors using a YAML manifest [Airbyte 1.6 Release Notes]. Directly uploading custom Docker images is primarily an Open-Source feature, and on Cloud it is only possible with a specific sales contract in place [Forum Post: API for uploading source plugin; Forum Post: Deploying CDK Connector]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements