Hi I deployed airbyte on docker i noticed that worker launches source containers and checks if config is correct. and worker is passing data like credentials to using named docker volume airbyte_workspace which contains config.json credentials are getting stored in docker volume even without encryption doesn’t seems correct is there any other way so that creds and config is passed from database instead using json file

Yes, there is a way to store credentials in a more secure manner. If you're deploying Airbyte Open Source on Google Cloud Platform (GCP), you can use Google Secret Manager to store credentials instead of in the database. Here are the steps to do so: 1. Create a service account with Google Secret Manager with read/write access. Generate a JSON key. 2. In the Worker and Server applications, set the

SECRET_STORE_GCP_PROJECT_ID

environment variable to the GCP project to which the credentials have access and secrets will be located. 3. In the Worker and Server applications, set the

SECRET_STORE_GCP_CREDENTIALS

environment variable to the JSON key created in step 1. 4. In the Worker and Server applications, set the

SECRET_PERSISTENCE

environment variable to

GOOGLE_SECRET_MANAGER

. Please note that this process is not reversible. Once you have converted to a secret store, you won’t be able to reverse it. For more details, you can refer to the [Airbyte Security Documentation](https://docs.airbyte.com/operator-guides/security).

@kapa.ai am using aws

That's great! AWS offers a wide range of services that can be integrated with Airbyte. Here are some guides on how to set up and use various AWS services with Airbyte: 1. AWS CloudTrail: Follow [these steps](https://docs.aws.amazon.com/powershell/latest/userguide/pstools-appendix-sign-up.html) to get your AWS access key and secret. [source](https://docs.airbyte.com/integrations/sources/aws-cloudtrail) 2. Deploying Airbyte on AWS EC2: This guide will help you deploy Airbyte on an Amazon EC2 instance. It includes information on the recommended instance types for testing and production environments, and how to create and download an SSH key to connect to the instance. [source](https://docs.airbyte.com/deploying-airbyte/on-aws-ec2) 3. Setting up Amazon S3 as the Airbyte destination: This tutorial guides you through the process of setting up Amazon S3 as your Airbyte Cloud destination. It includes steps on how to create a user group with Amazon S3 access and how to configure Amazon S3 as your Airbyte Cloud destination. [source](https://airbyte.com/tutorials/intercom-api-s3) 4. Setting up an Amazon RDS database: This tutorial guides you through the process of setting up a Postgres database as an Airbyte source using Amazon RDS. It includes steps on how to create a security group and an Amazon RDS database instance. [source](https://airbyte.com/tutorials/postgres-to-redshift-data-replication) 5. Setting up a Redshift destination: This tutorial guides you through the process of setting up a Redshift destination. It includes steps on how to create a Redshift cluster, modify the security group to allow inbound connections from Airbyte, and enable public accessibility from Redshift. [source](https://airbyte.com/tutorials/postgres-to-redshift-data-replication) 6. Setting up DynamoDB: This guide will help you set up DynamoDB as a destination. It includes information on how to fill up DynamoDB info and how to ensure your DynamoDB tables are accessible from the machine running Airbyte. [source](https://docs.airbyte.com/integrations/destinations/dynamodb) 7. Setting up S3: This guide will help you set up S3 as a destination. It includes information on how to prepare an S3 bucket that will be used as a destination and how to create new Access Key ID and Secret Access Key. [source](https://docs.airbyte.com/integrations/destinations/s3) 8. Setting up Amazon Seller Partner: This guide will help you set up Amazon Seller Partner as a source. It includes information on how to register an Amazon Seller Partner application and create an IAM user. [source](https://docs.airbyte.com/integrations/sources/amazon-seller-partner) Please let me know if you need information on a specific AWS service not mentioned above.