https://discord.cloudflare.com logo
Join Discord
Powered by
# r2
  • e
    try 3
    Copy code
    2022-05-01 18:43:19 (53.7 MB/s) - Read error at byte 4373286912/5883697152 (The TLS connection was non-properly terminated.). Retrying.
  • e
    you get the idea
  • v
    Yeah, for the total amount seems fine. The transfer rate is the surprising bit
  • i
    Huh... those numbers are contradictory.
  • i
    The 56.1 one has less progress than the 53.7
  • v
    "Fine" in the sense of in the ballpark of where I would think the limit is
  • e
    it seems to fluctuate quite a lot during the download
  • i
    4211154944 vs 4373286912
  • v
    The transfer rate and total amount transferred are counterintuitively decoupled
  • i
    Oh... interesting. Why is that?
  • v
    Network bottleneck vs CPU time to decrypt
  • i
    Oh, I see.
  • i
    Makes sense, although strange that it would tell you download speed of a file when it's not accounting for decrypt time.
  • e
    the VM used there is fully capable of at least gigabit when downloading from other sources, so i would assume the speed is coming from cloudflares end
  • v
    For sure
  • v
    Network throughput vs total CPU runtime used by R2 are separate things. The former is your download speed. The latter is how large a file you can download.
  • v
    Assuming no bottlenecks obviously but our decrypt throughput is ~140MB/s based on the total data transferred above. We should be ~10x that and we should have a strategy for downloading the full 5TB. Might have to update the limits piece to reflect the current state
  • d
    How can I connect r2 to rclone
  • j
    Search this channel for "rclone" - there's been a lot of discussion there. I don't think anyone has got it working yet though, seemingly due to https://github.com/rclone/rclone/issues/5422
  • i
    Huh, that looks like rclone is also trying to use a presigned request which R2 doesn't support.
  • e
    You can list and download files, but uploading is currently blocked because R2 doesn't support presigned requests
  • e
    Yes, that
  • e
    This was what we determined after a multiple day rabbit hole
  • e
    Lol
  • i
  • d
    What I can use then?
  • j
    Use for what? There's lots of ways to work with R2 today - from a Worker, the S3 API, etc. It's in private beta right now, so if you're not comfortable diving into things early, I might recommend waiting until it's more widely available and tooling has had a chance to catch up.
  • d
    I just Wana do daily sync to r2
  • d
    Kinna use r2 as a sync tool
  • j
    I believe folks have gotten the AWS CLI to work pretty well. I'd recommend checking the pinned messages and history in this channel, while stuff is still in beta
1...100101102...1050Latest