I migrated from S3 → R2 with Rclone

Then I messed something up, I tried rclone and I lost the Metadata, I'll try again thanks

What is the data resiliency like for R2? should we absolutely depend on R2 to not lose our data? obviously best practice would be to also have a backup somewhere like AWS Glacier, but I'm really curious. I haven't seen anything specific

Hey, I think I found a bug with R2 uploads. Trying to upload an empty

Blob

works but a

Blob

with any content at all fails & returns a

Network connect lost.

error. Anyone else tried this / encountered this?

ts
// Works
await env.MY_BUCKET.put('abc', new Blob())
// Doesn't work
await env.MY_BUCKET.put('abc', new Blob(['hello']))

Ah yeah that seems to have fixed it.

Slightly confusing as

Blob

is an accepted type but needs to be converted into a

ReadableStream

IM trying to use the Terraform example from the documentation. However when applying the configuration I get this error:

error creating S3 Bucket (prefix-preview-assured-moray-package-store): MissingRegion: could not find region configuration

I did configure the

aws

provider with `skip_region_validation`:

tf
provider "aws" {
  access_key                  = var.cloudflare_r2_access_key_id
  secret_key                  = var.cloudflare_r2_secret_access_key
  skip_credentials_validation = true
  skip_region_validation      = true
  skip_requesting_account_id  = true
  endpoints {
    s3 = "https://${var.cloudflare_account_id}.r2.cloudflarestorage.com"
  }
}

Did someone see that before?

Have you tried setting

region = "auto"

in the provider? I set it, among other things, following the provider configuration from https://github.com/cloudflare/terraform-provider-cloudflare/issues/1664#issuecomment-1204835324 and it worked in my case.

Btw. is there a way to have an R2 access token that is not tied to particular user/member but to the whole account/organization? Alternatively, is there a good way to create non-human users?

Yep that fixed it. Thanks 🙂

R2 access tokens under the hood are just API tokens, and those are unfortunately bound to users, with service/non-human users not being a feature that exists. The ability to create API tokens bound to accounts and not users is coming, but there is no ETA. > Building these API restrictions is a first step on the way to allowing account-owned API tokens (which will limit the need to have personal tokens)

In the interim I would recommend signing up for a new user, possibly from a shared email, and creating the API token using that user

Is there a fast way to bulk upload 22,000 tiny SVG files? Wrangler seems to take about 5-8 seconds per file which is wayyyy too slow. The dashboard only supports 100 at a time and I'd rather not manually upload 220 times. Is S3 any faster or does it also have a lot of overhead setting up the connections?

rclone

and the S3 API is probably more performant

Wrangler uses the dashboard API which isn't really meant for anything more than simple administration here & there.

ohhh that explains why it was so slow probably, I'll give S3 a shot

thanks!

rclone can do any amount of concurrent uploads which makes it super fast

Mm while uploading all of a sudden my jobs fail with an error:

500. Code: Internal Error, We encountered an internal error. Please try again.

. The last request returned

cf-ray=7ab0d43458a91c10-AMS

. Would you be able to tell me what is going on?

Two more:

cf-ray=7ab0d3a9dd250eaa-AMS

cf-ray=7ab0ed4a8f30b8c7-AMS

all the same error.

Are you still seeing this?

!kick 656570674625904643 Random weird ad

Kicked davidehuey#7766 (Case #1021) (user notified with a direct message)

This keeps coming up 😢 Is there any plan to fix this issue with r2 not respecting range headers when > 512mb? https://github.com/Chocobozzz/PeerTube/issues/5698

(Specifically https://community.cloudflare.com/t/public-r2-bucket-doesnt-handle-range-requests-well/434221)

Are you using an R2 public domain with PeerTube, or reading directly from R2 each time?

Looks like this isnt in any way an R2 issue but a cache issue. If you turn off cache it should work as described in the linked forum posting

is there a way to turn off cache for bigger files via rules to just skip over it completely and have it enabled for other stufff

Does anyone use R2 as part of your ML pipeline? If so, are you using it to take advantage of the zero egress fee? How does this help you?

Try https://dash.cloudflare.com/?to=/:account/:zone/caching/cache-rules/new