Hello I am trying to set up Airbyte on EKS and would like th Airbyte #feedback-and-requests

Hello, I am trying to set up Airbyte on EKS and wo...

Vikram Bhamidipati

02/26/2022, 2:35 AM

Hello, I am trying to set up Airbyte on EKS and would like the logging to go to S3. The CloudStorageConfig for S3 has a couple of issues that is making this hard for us. It expects an Access Key and Secret Access Key to be configured and is required per this. Also it uses

AwsBasicCredentials

. This is not allowed per our security policy, we need to use IAM roles or in case of EKS can use IRSA. Issue # 5282 addresses this in some way. I am wondering if this can be enhanced to use the DefaultCredentialsProvider. This will allow the logging to be more generic and work everywhere irrespective of how the AWS credentials are configured. Q - can I submit changes as a PR to this core functionality or does this need to be handled by the Airbyte Team ?

Vikram Bhamidipati

02/26/2022, 2:38 AM

Related question - Can I also submit PRs for the helm chart ?

William Phillips

02/28/2022, 5:59 PM

Hey @Vikram Bhamidipati, I think that your problem might be solved by this PR. You can of course contribute to whatever part of the repo you want by opening a PR from a forked repo, and helm chart contribs are more than welcomed!

William Phillips

02/28/2022, 6:41 PM

Hi @[DEPRECATED] Augustin Lafanechere: Thanks for your response. In this case, while the problem is the same - it is the area of Airbyte that differs. The PR you referred is related to the S3 connectors. The issue I have is with the core platform area of Airbyte. This is related to the Cloud Logging functionality of Airbyte. When it comes to logging to S3, the Code and the helm chart expect Access Key and Secret key only to use BasicCredentials. The code is here.

12 Views

Open in Slack

Previous Next