Use websockets within the DO to allow the request limit to be raised. Send a message with the parameters through the socket (you can batch them) and then receiver handles the downstream requests. Each message received resets the limit 🙂

I have just checked the dashboard. The errors started on Friday, January 13th at 12:00 am.

Hey @Robin it can probably be connected to a recent release but can't confirm because the team is out today. Can you give me your timezone for the team to check (tomorrow when they're back) if it's connected to the release?

Hi @Vero 🐙 . Thank you for looking into this. My timezone is CET / UTC+1

@Robin can you DM me your account id and the name of the script you're seeing the issue on?

I was just hit with a much larger than expected bill, suggesting crazy high DO runtime... yet looking through each project, and each DO, I can't find one that could even be close to being the culprit. Is there a way to diagnose where the spend/compute time is coming from??

Dramatic rise in

Network connection lost

in the last 10 minutes or so...

Looks like there is a pretty big spike in these errors for me as well from a quick look at my error logs filtering down to "Network connection lost" In the last 3 hours I had 3103 of these errors and filtered to the last 30 days the total is 4814 which means basically all of those errors are in the last 3 hours.

eyeball failures to call region-specific DO (colo in do name) in the last hour, by eyeball colo:

tsv
MEL    377
SYD    279
SIN    148
BOM    36
CHC    20

still ongoing, rate increasing actually

all DOs for those eyeballs would be in

SIN

according to where DOs live

(I see similar results SIN seems to be having some trouble with DOs)

Yeah we're having an issue, status being updated shortly

How are things looking for you at this point

less red

never seen so many of those before (at least since sept)

zero in the last 15 minutes!

thanks!

we're monitoring but feel free to DM me if you see it creep back up

here's a new one:

Error: Durable Object namespace binding cannot communicate with other nodes.

How does this differ from network connection lost and the other existing error states? I think I'm getting this from eyeball -> DO, not DO -> DO

@john.spurlock there was a single machine in a misconfigured state, should have stopped within half an hour of you posting that message

does that mean the eyeball can't use a single stub, or it can't use any stub? I'm curious if I could retry in this state by sending to another fallback DO in another region - or if that eyeball just can't talk to DOs anywhere

Do you do DO DO requests?

yes

But I think the

Durable Object namespace binding cannot communicate with other nodes.

was being thrown only in eyeball

Hmm, my hunch would be that it was thrown from a DO trying to call another. But to answer your question is means any client on that machine can't make a stub. It should really never happen, but here we are 😦 That one machine was somehow not fully provisioned

Hi I'm trying to make a scalable layer between Whatsapp and an API. For each message I receive I need to get the "profile" of the one who send it, which is under 128KB. So what I don't fully understand is why I need seperate namespaces per Durable Object. Can I not just make one big Durable Object that will save all profiles? Any suggetions?

A single object can only handle so much load (baseline is generally around 100rps). So if you expect to get more than 100 messages a second you will need to have multiple objects so that you aren't overloading it.

But having multiple objects means I also have to duplicate all profiles per DO?