https://linen.dev logo
c

Cristiano Sarmento

10/15/2021, 2:49 PM
Hello, I have a doubt about setting up an S3 source, is it possible to use an assumed role to connect instead of specifying the secret Id and Secret? Lets suppose my airbyte is in an EC2 instance that has all the rights to an S3 private bucket via Assumed Rule, is it actually possible to connect this way by leaving the Secret Id and Key blank ? Thanks!
u

user

10/15/2021, 3:15 PM
You may want to follow this ticket. https://github.com/airbytehq/airbyte/issues/5942 The feature isn’t available yet. but we know a number of people would like this.
u

user

10/15/2021, 3:35 PM
For k8s I can understand - but for Docker running on an EC2, why is instance metadata not passed through to the containers? This would just work in this case
u

user

10/15/2021, 3:36 PM
You’re welcome to leave follow-up comments on the issue. That helps us hear what people need and keep track of that (not buried in slack) when we start work on the ticket.
u

user

10/15/2021, 4:38 PM
So I've actually just tested this with my SQS source https://github.com/airbytehq/airbyte/pull/6937 and assumed roles through EC2 docker works without any change (to the env) - it seems to correctly inherit the instance metadata
u

user

10/15/2021, 4:40 PM
I would imagine the same would apply to the s3 source, I will test it now..
u

user

10/15/2021, 5:04 PM
So it doesn't work for source-s3 - but this must be down to how source-s3 is using boto to build the s3 client and not due to the metadata being available
u

user

10/15/2021, 5:13 PM
Thanks for testing stuff. Adding your notes to the issue tracker would be great, too.
2 Views