https://discord.cloudflare.com logo
Join Discord
Powered by
# r2
  • v

    Vitali

    05/05/2022, 12:01 AM
    I think what you're talking about is multipart
  • v

    Vitali

    05/05/2022, 12:02 AM
    That's the only "chunking" thing related to large files
  • v

    Vitali

    05/05/2022, 12:04 AM
    https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
  • h

    HardAtWork

    05/05/2022, 12:22 AM
    Correct me if this isn't relevant here, but I was thinking(if possible) that a client would send a chunked file to my domain. The domain has a Worker on it, that reassembles the file into a ReadableStream, that can then be piped into R2.
  • h

    HardAtWork

    05/05/2022, 12:23 AM
    I seem to remember that this was what FlareDrop did before albert pushed the new update.
  • a

    andrew

    05/05/2022, 12:34 AM
    has anyone done throughput testing on R2 from a source that has plenty of downstream bandwidth (>= 10Gb/s), using simultaneous ranged GET requests against a large file across multiple client threads?
  • a

    andrew

    05/05/2022, 12:34 AM
    curious what folks are seeing so far throughput wise when you really push the pedal down
  • c

    Craiggles

    05/05/2022, 1:22 AM
    Nowadays, if you’re using fetch, you’re getting the best results you can ask for (in other words don’t use XMLHttpRequest). Also, cloudflare uses http3 so that’s another massive bonus. 9/10 the limitation is the clients upload limit from their service provider imho.
  • a

    andrew

    05/05/2022, 1:34 AM
    hm. i'm not talking about browser fetches
  • e

    Erwin

    05/05/2022, 1:52 AM
    Hehe.. I am pretty sure @Vitali has been doing that a lot lately.. as a large part of his current work in improving that.. Found some interesting bottlenecks that way.. I think currently decryption is the largest bottleneck?
  • a

    andrew

    05/05/2022, 1:55 AM
    mmm yes, hopefully you're using something backed by the silicon there, like AES
  • a

    andrew

    05/05/2022, 1:56 AM
    (intel has AES-NI, arm has their equivalent i'm sure)
  • v

    Vitali

    05/05/2022, 2:19 AM
    The decryption piece I've benchmarked at 100 MB/s. In the next release I got that up to 400 MB/s but I'm not convinced encryption/decryption is the bottleneck today (someone was reporting 50 MB/s over a 1Gb/s link which is ~2x less than what you would expect). 10 Gb/s is going to be much trickier to get to. Certainly not a goal for GA.
  • v

    Vitali

    05/05/2022, 2:22 AM
    If by "chunked" you mean "multipart" then you could do that although I don't understand what the reassembling piece would be doing. If by "chunked" you mean do a "chunked HTTP transfer" you could do that but you're going to have to know the length up-front and somehow transfer it side-band to properly instantiate a
    FixedLengthStream
    . Chunked transfer here is irrelevant though - at the Worker level the
    ReadableStream
    body you get is already "dechunked"
  • h

    HardAtWork

    05/05/2022, 2:25 AM
    Thanks for that explanation. It still does fully make sense to me, but that just means I need to do some more reading.
  • a

    andrew

    05/05/2022, 2:31 AM
    400MB/s is pretty respectable
  • v

    Vitali

    05/05/2022, 3:16 AM
    Actually HTTP/3 is not turned on for the R2 zone and we have an AI to turn off HTTP/2 by next week (we had an internal user that assumed we'll be HTTP/2 so we're holding off for them). For those that inevitably ask "why", the answer is "reasons".
  • c

    Craiggles

    05/05/2022, 3:18 AM
    lol nooooo
  • c

    Craiggles

    05/05/2022, 3:20 AM
    honestly that has to hurt performance thou doesnt it? You're just sending everything as plaintext
  • c

    Craiggles

    05/05/2022, 3:20 AM
    also no multiplexing
  • v

    Vitali

    05/05/2022, 3:23 AM
    HTTP/2 multiplexing generally isn't that useful due to head-of-line blocking of TCP. HTTP/3 it gets more useful. However, in the S3 API case what's the benefit? The entire spec is built around opening up separate connections for parallel transfers. Workers itself only really supports HTTP/1 anyway so HTTP/2 and HTTP/3 are going to have an impedance mismatch and need an extra translation layer.
  • v

    Vitali

    05/05/2022, 3:34 AM
    Newly deployed release (will take a few min to go live): * The empty string and
    us-east-1
    are now accepted as region parameters. They automatically alias to the "auto" region. This also applies to the location constraint for
    CreateBucket
    . *
    GetBucketEncryption
    ,
    PutBucketEncryption
    , and
    DeleteBucketEncryption
    is now supported (the only supported value is AES256). Additionally, we now correctly reject most bucket-level S3 actions as explicitly unimplemented instead of treating it as `ListObjectsV2`/`PutBucket`/`DeleteBucket` respectively. * S3: Fix handling escaped entities being sent (e.g. rclone CompleteMultipartUpload).
  • v

    Vitali

    05/05/2022, 3:40 AM
    That's just for the raw encrypt/decrypt piece. I don't have good insights into if there's other bottlenecks elsewhere.
  • v

    Vitali

    05/05/2022, 3:42 AM
    Also, if y'all haven't noticed, the API tokens can be created from the bucket settings & has the correct tokens you'd paste into your favorite client (no more sha256sum). This UI will probably see a little bit more iteration (hanging off the bucket is a bit awkward) but hopefully can make onboarding a tad easier
  • c

    Craiggles

    05/05/2022, 3:42 AM
    I was thinking about get requests mostly
  • c

    Craiggles

    05/05/2022, 3:43 AM
    although, once its cached, workers will still use http3?
  • v

    Vitali

    05/05/2022, 3:44 AM
    I promise you there's no benefit for R2 currently to use HTTP/3 and definitely no benefit for HTTP/2
  • v

    Vitali

    05/05/2022, 3:44 AM
    Issue the requests in parallel over HTTP/1 connections. TLS fast starts should minimize the cost of establishing a connection
  • c

    Craiggles

    05/05/2022, 3:45 AM
    I thought the whole concept of treansfer acceleration used by aws was a lot like http3
  • c

    Craiggles

    05/05/2022, 3:45 AM
    but i could be getting this all mixed up
1...115116117...1050Latest