Another release thanks to bugs found in rclone test suite (thanks @ncw): Fix problem where copying across buckets from a key that doesn't exist or from a source bucket that doesn't exist results in a confusing "unimplemented" error message. Not sure if that fixes the issue but hopefully error messages are at least better now. S3 responses now have content-type correctly set to application/xml for endpoints that return XML. Multipart uploads now have a -N suffix on the etag where N represents the number of parts the file was published with. Improve error response of UploadPart/UploadPartyCopy when there's too much concurrency or given an upload id that no longer exists (instead of returning an opaque "internal error" message). UploadPart can now be sent a 0-length part.

ListObjectV1 hopefully in tomorrow's release

+ more bug fixes

so do we get charged

$0.36

per 1 million file downloads?

(+ 10 million for free)

Yes.

Great work! I just discovered if you don't supply a

region

then the aws SDK sends an empty body which makes creating buckets not work. I'll give the integration tests a whirl now

2022/05/06 11:32:32 DEBUG : HTTP REQUEST (req 0xc000304400)
2022/05/06 11:32:32 DEBUG : PUT /rclone-test-jozibay2rojeveb1yejakiq4 HTTP/1.1
Host: 14aad7c9ed489151b51557e321b246cf.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.0-DEV
Content-Length: 0
Authorization: XXXX
X-Amz-Acl: private
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220506T103232Z
Accept-Encoding: gzip

Response

2022/05/06 11:32:32 DEBUG : HTTP RESPONSE (req 0xc000304400)
2022/05/06 11:32:32 DEBUG : HTTP/2.0 400 Bad Request
Content-Length: 149
Cf-Ray: 70710d741f867509-LHR
Content-Type: application/xml
Date: Fri, 06 May 2022 10:32:32 GMT
Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
Vary: Accept-Encoding

<Error><Code>MalformedXML</Code><Message>The XML you provided was not well formed or did not validate against our published schema.</Message></Error>

I've run the basic integration (backend) tests for rclone - they are working much better now 🙂 All tests passed except - Range requests with an open end (which you know about already) (**update**: I discovered the code which is supposed to make Range requests absolute wasn't working in this case, so I fixed that and now this bit of the integration tests pass! It is still an R2 bug but it no longer matters to rclone!) - Tiering tests (moving objects between tiers) which I don't think you support So looking very good! I'll run the full integration test suite now and see what happens - it might throw up a few more oddities.

Yeah R2 doesn't have any tiers at all so I'm not surprised that would confuse it

I don't have access to R2 yet but I was wondering about consistency. If I update a file in a CRON trigger and then I get it for a HTTP request 1s later (in a different colo) then will I get the updated version?

Or will it take a few seconds to propagate? (like KV can take up to 60s for changes to be seen in other colo's)

Search for

strongly consistent

in this channel - there's lots of good answers. KV is eventually consistent whereas R2 is strongly consistent. https://discord.com/channels/595317990191398933/940663374377783388/963515574187622420

Oh thanks 👍

To interact with R2 in majorities of scenarios (except CF dashboard, and R2 S3 API,...), you need to bind with a CF Worker, which is part of the pricing, but it is not mentioned in the docs or well advertised. KV is well documented with regards to that condition.

It would have to be carefully documented if so, since there are many use cases that involve using the S3 API and thus don't incur worker charges KV has an API too, but its far less obvious or useful to what most people would want to do, so that's probably why KV has workers documented as part of pricing and not R2

If its just adding an extra note you can probably PR the docs and im sure they'd be happy to see it

> KV has an API too Wonder if there's people who purely use

wrangler kv

haha

No Worker, just that.

You can use https://api.cloudflare.com/#workers-kv-namespace-create-a-namespace

and more

$NET Q1 earnings call / Matthew Prince: [...] __R2 is going to progress from a closed beta to an open beta next week.__ And next week, we have Platform Week, which is really very much focused on all of the different ways that Cloudflare can be a platform for developers to develop on, but really a focus around the various Workers' products. And I think we'll have some very exciting surprises as part of what we're rolling out over the course of the week as well. [...]

I will refrain from PRs on R2 subject, until it is mature enough, or at least GA, yesterday, I submitted a PR, to later know that the problem is CF dashboard, the Quick Editor not assuming a compatibility date, not even considering today's date. The most basic docs are needed and methods to know how to interact. We can get them here https://github.com/cloudflare/workers-types/blame/master/index.d.ts But changes can happen without any notice, and then documentation goes wrong again.

Here is the result of the full integration test suite: https://pub.rclone.org/r2-integration-test/ Only 4 failures now which is an excellent result - well done 🙂 Of the failures, -

TestIntegration/FsMkdir/FsPutFiles/ObjectOpenSeek

- is Range failure as before, but shouldn't be there still - me to fix -

TestIntegration/FsMkdir/FsPutFiles/SetTier

is expected -

TestMultithreadCopy/{size:131073_streams:2}

is a new one - see below -

TestCopyFileMaxTransfer

is a left over from my

PutObject

patch - me to fix The new failure is rclone trying to read the last byte of a file using a Range request so I think there is an off by one in the

Range

checking;

Ranges

are (inclusive,inclusive) not (inclusive,exclusive) like everything else in the entire world which must surely be a cause of a lot of bugs! This is reading a 131073 byte file

2022/05/06 13:46:49 DEBUG : HTTP REQUEST (req 0xc000a4a000)
2022/05/06 13:46:49 DEBUG : GET /rclone-test-yuqaten6kulabay0lurevex8/file1 HTTP/1.1
Host: 14aad7c9ed489151b51557e321b246cf.r2.cloudflarestorage.com
User-Agent: rclone/v1.59.0-DEV
Authorization: XXXX
Range: bytes=131072-131072
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20220506T124649Z

Response

2022/05/06 13:46:49 DEBUG : HTTP RESPONSE (req 0xc000a4a000)
2022/05/06 13:46:49 DEBUG : HTTP/2.0 400 Bad Request
Content-Length: 109
Cf-Ray: 7071d2257c738926-LHR
Content-Type: application/xml
Date: Fri, 06 May 2022 12:46:49 GMT
Expect-Ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare

<Error><Code>InvalidArgument</Code><Message>Invalid Argument: range range must be positive.</Message></Error>

This is easy to replicate with

rclone cat -vv --tail 1 --low-level-retries 1 --retries 1 r2:rclone/file.txt --dump bodies

Here is the latest rclone with all the latest patches and a Cloudflare Provider. This works very well in my tests - please have a go with it! https://github.com/rclone/rclone/issues/5642#issuecomment-1119634829

This is awesome!!

omg so cool seeing you here Nick! Been using rclone for years, love the work you're doing!!

The fix for that should be going out shortly

Same fix as the previously reported

0-0

range problem.

(I think - please let me know if not the case)

wow!! this is so exciting