Hi all, I am trying to setup airbyte on AWS EKS Cluster and I want to use S3 as log location I have configured AWS IAM User Credentials and Log bucket name and region as well and I removed minio because I am gonna use S3 for storing logs but due to this airyte-worker is not coming up and when I go with default minio deployment then everything works fine. I followed airbyte document but it doesn't work for me. I don't know what I am missing. If anyone can help me that would be great

Is there an error stack trace corresponding to the issue

Screenshot 2022-11-16 at 3.55.53 PM.png

Are you deploying using Helm? If so, I probably know what the issue is.

I am not deploying using helm

My hunch is that Airbyte has a bug that isn't passing custom S3 creds through to the worker for some reason in EKS. No idea why. Try explicitly setting the environment variables on the worker and see if that fixes it. That's what it was when I was getting that same error, and I fixed it the helm way.

@Ethan Brouwer have tried that but still the same error exist

@Vikas Goswami to double check, were you following this doc? https://docs.airbyte.com/deploying-airbyte/on-kubernetes/#configure-the-custom-s3-log-location We already have an issue open for this too https://github.com/airbytehq/airbyte/issues/14576 I'm looking into this more!

@Nataly Merezhuk (Airbyte) I have double checked everything but didn't find anything suspicious and yes I am following the same doc which you have shared. don't know what cause the issue ?

What version of Airbyte are you on? Is it lower than

0.40.8

?

It's

0.40.10

and I had tried with

0.40.18

also

I'm also having the same issue on 0.40.18

@Gabriel Martelloti I am deploying it from kustomize

Here's something to debug what's going on (if it was a similar issue to what I had). Run

kubectl describe pod <name of worker pod>

Check the

Environment

and look for

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

and check that those are set. If they're set via a secret, then go check the secret to make sure they're set correctly. The issue stemmed basically from the access key and secret being empty, which threw the exception.

Previously, it was set from secret. I have changed and set them directly inside the deployment. It is correct

Let me look into this further, sorry to hear that none of these suggestions are working for you!

yes please

@Vikas Goswami, are you using IRSA?

@Nataly Merezhuk (Airbyte) yes we are using IRSA and I have already checked this issue didn't get any help

@Nataly Merezhuk (Airbyte) We’re also experiencing a similar issue with

kustomize

+ custom S3 logging. I’ve also stated my issue here as well. Would appreciate any clues!

So it looks like this is a known issue, the one I linked above: https://github.com/airbytehq/airbyte/issues/14576 I'd say the best course of action is to comment on it and help it gain visibility. It's already triaged, so the engineering team should have it in their backlog. Issues like this are not prioritized, unfortunately, but if somebody has time submit a PR - please do so! That's why we're open source after all :)

hi @Nataly Merezhuk (Airbyte),Is this problem solved? I also encountered this problem when using kubernetes+custom S3 to deploy airbyte.

The issue on GitHub is still open, that means it has not been resolved.