https://serverless-stack.com/ logo
#random
Title
# random
a

Adrián Mouly

05/24/2022, 5:05 PM
Hey guys, Is there any recommendation on how to organize S3 buckets? for example, multiple buckets, one for each feature, or subfolders by service, .. etc? Following the micro-services model, we create a bucket for each micro-service (in our case, we have services running on k8s). But also, I wonder if CDK can define properties to “backup” those buckets?… Want to hear ideas from you guys 🙂
f

Frank

05/24/2022, 5:08 PM
I ran into S3 api call throttling issue a couple of times. (It’s a soft limit that you can request to get lifted.) Since then, I’ve tried to go w/ 1 bucket per feature.
a

Adrián Mouly

05/24/2022, 5:13 PM
Nice.
s

Seth Geoghegan

05/24/2022, 5:13 PM
also keep in mind any authorization requirements you'll have. You can use the bucket keys in your security policies e.g.
/uploads/<user_id>
a

Adrián Mouly

05/24/2022, 5:13 PM
Regarding bucket backup… I was looking into this, https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_backup-readme.html
I like that, because I was wondering about how to organize the folders… if I need them or not.
But also, the way I access the files is using the “pre-signed url” thingy.
s

Seth Geoghegan

05/24/2022, 5:15 PM
pre-signed url's have the access permissions of the creator of the url
a

Adrián Mouly

05/24/2022, 5:15 PM
So I never provide access to files directly, it’s always api-driven, and then user can hit the file (actually we use CFront signed).
s

Seth Geoghegan

05/24/2022, 5:15 PM
which, might work perfect for your use case
a

Adrián Mouly

05/24/2022, 5:15 PM
Yeah I have to think about it.
I started to pay attention to this due today somebody decided to modify the “expire” time of a
/temp
folder that we have in Production. Apparently the files there weren’t supposed to be temporal 😂 .
So… sometimes developers doesn’t pay attention to infrastructure configurations/setups… and they just “put” a file.
s

Seth Geoghegan

05/24/2022, 5:20 PM
I've seen the exact same thing
it's super easy to get Bucket permissions wrong
a

Adrián Mouly

05/24/2022, 5:20 PM
Yes! totally.
s

Seth Geoghegan

05/24/2022, 5:20 PM
I've seen a single
uploads
directory that contained uploads from multiple customers in a multi-tenant system. The bucket was not public.
a

Adrián Mouly

05/24/2022, 5:21 PM
And also, people forgets about this, they just create a bucket when the project starts… and it just a resource that is never checked again while the application works fine.
s

Seth Geoghegan

05/24/2022, 5:21 PM
However, every object within the bucket was public
a

Adrián Mouly

05/24/2022, 5:21 PM
Yeah, we have that problem here too.
That’s why I’m working on separating buckets by feature/platform.
s

Seth Geoghegan

05/24/2022, 5:21 PM
so, nobody could list the contents of the bucket, but anyone could get a file if they simply guessed the object name
do we work at the same place? 😆
a

Adrián Mouly

05/24/2022, 5:21 PM
Hahahaha.
I think this is normal, starts with 1 bucket when the app or team is small.
But suddenly you have 5 teams and 20 developers, all using the same bucket.
k

Klaus

05/24/2022, 5:27 PM
Regarding files being public by accident - cloud front OAI is a nice way to prevent it
Cloudfront - Origin (select Origin) -> edit
Then block public access can be enabled in S3 permission (or via code;) )