Which is probably why that fix is wrong

The generated request is probably fine for presigned requests but I haven't skimmed that spec yet

fair enough, i honestly didnt realise because i dont know a lot about how S3 works

and i was too tired right now to realise that one was query and not header

No worries

@john.spurlock what’s the concurrency of the s3 cmdline multipart upload? Is there some way to limit that?

Try setting max_concurrent_requests to 2 or 3. Default is 10. That’s likely the problem

not sure if you were joking or not - but imo vhost is synonymous with s3 because you can cname to it and (with the help of public-read and presigned urls) made (and still makes) s3 a poor-mans one-region cdn. the actual bucket name is a trivial implementation detail and, since it's a shared namespace on amazon, usually can't get nice names anyway i did not realize that amazon actually announced that they were going to remove path-style, and it's currently on borrowed time https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

I had

max_concurrent_requests

at 4. Lowering it to 2, re-enabling multipart by setting

multipart_threshold

to a sane value seems to work with r2! Unfortunately this concurrency setting also prevents aws s3 sync from processing multiple files (more than 2) in parallel as well. The current r2 restriction on 2 parallel requests is only per multipart upload (upload id) right? Like I could upload 10 files to the same bucket at the same time as long as each individual multipart job did not exceed 2 simultaneous requests? potentially 20 simultaneous requests to r2 in this case

You can get up to ~200 writes/s to unique uploads before a bottleneck shows up (although I haven’t tested concurrent multipart). I’m investigating that one.

But you actually have that today with workers. Sure there’s rough edges to sand off to make it more no code, but isn’t a worker bound to a single bucket the same thing? Why couldn’t the s3 interface even be the same thing? Like you could bind the s3 frontend to a bucket or buckets on your unique zone.

and is that 200 uploads at a time per-bucket? per-account?

Next week hopefully fully per bucket. Today your bucket shares that with other buckets in the same colo (buckets in the same region are still in different colos)

the r2-wide limit is higher because writes from one bucket cannot saturate our metadata storage

would be outstandingly awesome if r2 responses had the storage colo + region in x- response headers

We haven’t agreed on how to think about regions/what regions mean/etc. colos and regions though is not how you should think about the product though. What happens when we start replicating the metadata globally? Any assumptions you have break.

yea just interesting for debugging / logging / mapping etc

We might start thinking about returning the Colo/region in a few years once we have the product flushed out more

so i noticed the single PutObject entity too large issue is fixed?

now i'm getting

503 Worker exceeded resource limits

somewhere between 1gb and 2gb single puts

I understand. But Hyrum’s law is the devil

Ugggh. I was hoping it would be higher.

regardless of payload signing (thought maybe i could squeeze out more cpu without payload signing)

I need to swap out the encryption

ah

Yeah payload signing will slow things down

Maybe. Hard to tell

I don’t think it’s the current bottleneck

I reserve the right to say totally contradictory things