R2 Access Via Domain is Down?
# workers-help
c
Howdy! I got a report from some of my users that R2 is no longer serving content. Everything from my custom domain is returning a 200 with 0 bytes (even if the file does not exist). For example: https://cdn.btst.io/test.html
Copy code
curl -i https://cdn.btst.io/test.html
HTTP/2 200
date: Thu, 11 Jan 2024 02:34:45 GMT
content-length: 0
access-control-allow-origin: *
vary: Origin
access-control-allow-credentials: true
access-control-allow-headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=8OA78wCnvcRLW8CC5ljmtce%2BUhoPFbwNekVtEOQFJR2pMlYT5PVokesuCbId8U2u05wrqi3%2FkxP3kUy3K92R%2BvEg3w7XGcMk5vQraiHBgoKxbYMbUfBYchbrM4woyQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8439c3330d605cae-RDU
alt-svc: h3=":443"; ma=86400
I don't know how long this has been happening, but I don't see any evidence reported anywhere? Is this an issue on y'all's side?
For what it's worth, workers fetching from R2 seem to be working fine. I would hate to rearchitect during downtime, but that seems like a viable path at this point.
k
I can't reproduce this with a custom domain, and the fact it also serves a
200 OK
on
/
despite
/
having never worked on custom domains is weird.
What is the CORS config on that bucket?
I suspect this isn't actually hitting R2, especially since it's serving CORS headers without me passing an Origin header.
c
I just swapped cdn.btst.io over to a worker, so it's working now. cdn2.btst.io is pointed at the bucket now and still showing the weird behavior
Copy code
✗ curl -i cdn.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:17 GMT
Content-Type: text/html
Content-Length: 25
Connection: keep-alive
ETag: "68219c28daba0ca0e67295f1d71aaf37"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=x4xcZfVXl5sGxXIIlP3uehMNGl7SfVRPkBu%2BF9WPODYldFIi%2FaGDLfUra4%2B6WjiF6nL%2F3B184A3k9pUj2cacA6nFS1mKa%2BkZ84y8jfpkg8%2Fx8tGRsMbTsn7hwyUn0w%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea1278115cb1-RDU
alt-svc: h3=":443"; ma=86400

<html>
Success!!
</html>
✗ curl -i cdn2.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:22 GMT
Content-Length: 0
Connection: keep-alive
Access-Control-Allow-Origin: *
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ioFpioSLw6%2B4yTuBiRiId5XFaKbmupwDEngFurqI00Psx6r77PrA68uWDly8vNZSbBuqTfIZpRsTaoOZttQMCoqP0REN3ytXbH4w1KKsckwtB791dE8qq3QuQ6Z0PGg%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea2cc8ed5ca1-RDU
alt-svc: h3=":443"; ma=86400
Yeah, super weird issue haha. It was working up until tonight though haha
k
See what says, you'll have the trace tool on the sidebar in the dashboard.
c
That's a cool little service!
Same response for the working one with a worker, FWIW
This still seems to be affected. I just realized that my less popular sites have broken R2 bucket CDNs as well 😞
k
@Sid | R2 might be able to help - I don't think the traffic is hitting R2, but I also don't have the ability to figure out why it isn't going to R2.
c
Thanks! That'd be helpful!! Am I asking in the right place? Or should I escalate / push somewhere else / wait for Sid? I am not super confident that I found the proper channel 😅 I'm also quite surprised that nobody else has reported.
k
I would be surprised if it's affecting anyone else but you.
I can't repro it and there's no other reports on the community forums or here
You'd probably be better off posting in #940663374377783388
c
Ah good to know! I wonder if a newer worker route is matching the cdn urls
Ugh. Indeed. I had a worker route added that started trapping requests before they made it to R2 😭 It's so interesting to me the way that worker routes are global across all subdomains.
Anyways, thanks for your help @kian !!
k
*/*
would be expected to match everything
You might want to scope it down to a more specific hostname
c
Oh yeah, I knocked it down. That originally came from some experiments with the "SaaS custom domain routes" for workers
k
Unfortunately the trace tool you used earlier doesn't yet tell you if a Worker route was matched
c
You kinda need a wildcard to catch those routes for whatever domains a SaaS user puts in and you want to run your worker on
Which basically means you need a dedicated TLD per function to run a SaaS w/ custom domains on. Unless I'm missing something or you add a route for each domain
c
You can do
*btst.io/*
, Service: none to stop it from taking over subdomains for that domain, the more specific one wins docs: https://developers.cloudflare.com/cloudflare-for-platforms/cloudflare-for-saas/start/advanced-settings/worker-as-origin/
c
Thanks @Chaika ! Is there something like a "not match" blocker, so I can do a
*
that matches SaaS domains, without matching domains on my TLD? In my testing SaaS domains (i.e. user provided domains), I needed a matching rule defined in order for my worker to serve traffic to them
c
Right, like the tutorial there says, you can make one more specific like
*.<zonename>.com/*
service none along with the
*/*
for SaaS, and the more specific one wins (so service none will apply to your own domain)
c
Ohhh I missed that nuance! That "None" is super helpful, thanks!