Hey All 👋 I feel like ive asked this question in multiple increments so maybe its best if i just summarize the question in one post. So what I am trying to do is create S3 as a destination. The method now is to create an IRSA that inherits the permissions to access a specific bucket. This is linked to the OIDC provider that already exists. The next step was creating a PR in out forked airbyte reppo to link the pod and pod serviceaccount (airbyte-admin) to OIDC. Effectively if airbyte supports irsa this should work. My questions are: • I am running on

0.35.28-alpha

which asks for non-empty values for AWS Access ID and secret Keys. Is this version capable of supporting IRSA for S3 as a destination? • What deployment is making to call to S3? I would think it is the

airbyte-worker

. • Our prod instance is failing to connect to the S3 logging bucket. Would that be configured the same way? Ie allow access to the logging bucket to the K8s servuce Account. Thanks in advance!

Hi Kevin! Thanks for summarizing! Could you also consolidate all previous discussions in a comment on this thread? That would be super helpful for future reference :) It really helps the community assistance team if there is one thread per question, so I will close the other discussion form yesterday: https://airbytehq-team.slack.com/archives/C021JANJ6TY/p1666636352314469 The issue you referenced is still open, the PR referenced in the issue has been closed but not merged: https://github.com/airbytehq/airbyte/issues/5942 So currently the S3 destination connector does not support IRSA, unfortunately. This might be the PR that implements it: https://github.com/airbytehq/airbyte/pull/17334 Let me look into the other questions and I'll get back to you! It's unadvisable to run on such an old version of Airbyte though as updates are merged very frequently and you will likely run into breaking changes. Is there a reason you're running that version of Airbyte?

interesting i saw this comment - https://airbytehq.slack.com/archives/C02MHKUE4BC/p1657543452260689?thread_ts=1657531229.198279&cid=C02MHKUE4BC and thought it was possible

I can't open the link for some reason, could you attach a screenshot?

Screen Shot 2022-10-25 at 11.23.57 AM.png

I haven't found a clear answer to this as of yet, could you test it out and see if it works?

so far it does not, because it requires a non empty value for both the key and secret

No problem! I've asked about this internally, but haven't received any new information yet. Since it's not working for you, my assumption would be this functionality is not currently available. Sorry about that! I'll keep looking, but meanwhile keep an eye on this PR: https://github.com/airbytehq/airbyte/pull/17334 Looks like that will implement roles and allow empty values for the key and secret. Looks like it's really close to being merged too, the user just needs to sign the CLA!

thanks! Since it is time sensitive, would it be possible to checkout this branch and port over the code to my local setup and use? @Nataly Merezhuk (Airbyte)

That is totally up to you, go for it!

also. do we know which deployment needs access to S3?

I don't quite understand your question, could you please elaborate?

kubectl get deploy -n airbyte
NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
airbyte-db            1/1     1            1           6h56m
airbyte-minio         1/1     1            1           222d
airbyte-pod-sweeper   1/1     1            1           222d
airbyte-scheduler     1/1     1            1           222d
airbyte-server        1/1     1            1           6h56m
airbyte-temporal      1/1     1            1           6h56m
airbyte-webapp        1/1     1            1           6h56m
airbyte-worker        1/1     1            1           6h56m

these are the pods that were deployed to K8s . I am trying to create an IAM role that connects to an existing OIDC provider. I would like to connect the pod to the Service account (airbyte-admin) that will inherit permissions to query S3. (see diagram attached). So i guess im wondering which of these deployments are hitting S3. that way we can patch the service account to that pod.

I got some input from a colleague and it looks like it should be the worker pod because a k8s deployment follows the same architecture as a regular docker deployment: https://docs.airbyte.com/understanding-airbyte/high-level-view/ And if that does not work, you might need to play around with the config: https://docs.airbyte.com/operator-guides/configuring-airbyte/#kubernetes-only

thank you!!

Hi Kevin, it's not clear to me what your asking, could you please rephrase the last sentiment/question?