I would retry immediately. Much more frequently that it doesn't make it to the DO at all. A big source of these was fixed just yesterday, and I have a few more on my plate up next, these errors anger me greatly. 🙂

Oh, so happy to hear that - this has really been the only point of flakiness seen in practice using DOs, otherwise rock solid

Hi there! I've built a prototype websocket server on workers which is working really well and now I'm wondering about scalability in terms of max client connections per DO. Are there any known limits at this point?

It really depends on how much work your DO does per message, since it's limited to a single thread. I couldn't give any guidance number-wise right now. We should probably figure out what the max a simple Websocket echo server could handle...

The work itself is fairly minimal, mostly just passing data from the "host" connection to all other connections. The messages are flatbuffers-encoded and it does a check on their type to determine if it should cache certain ones or not, but that's it beyond straight pub/sub. I can probably test myself to see how many clients, but wondered if any limits were already known.

If there are, I'm thinking we may be able to get around that by creating secondary workers that the listeners connect to which then connect to the main DO instance and pass messages on.

500ms per request. What is a request in a websocket world? If there are outbound connections, do incoming messages on those count?

Yes, this should perform fine. Let me know if it doesnt.

Just a bit of clarification: 1) Yes, DOs suspend a few seconds after they stop receiving requests. A DO that is continually under load will remain active, assuming it respects memory limits, there isn't a bug in the runtime, etc 2) Today, you are billed both for the connecting Worker (Request + duration (duration is only charged on Unbound)) and the DO (request + duration + storage operations). With websockets, your connecting Worker will continue to bill duration as long as the websocket is connected - we're looking to change this soon. 3) When a DO is suspended, it stops billing duration. If you haven't stored any data durably, you will have no charges. If you have stored data, you'll incur the stored data charge until that data is deleted.

You're given 500ms when the request is first established, and an additional 500ms per message.

Do you care about the singleton nature of DO here? I'm guessing by your second message you don't, and just want to be able to connect a large number of websockets to a number of DOs?

mind shooting me an email to gmckeon@cloudflare so we could chat a bit more?

So inbound messages count as requests? Cool. 🙂

I'm having trouble with setTimeout. The more significant one is the Error

You have exceeded the number of timeouts you may set.

It appears to kick in after only 10,000 iterations. The other one is

Cannot clear a timeout created in a different request context

which was annoying but relatively easy to work around. How can I raise the limit of 10,000? I hope that is not a fundamental restriction?

We're using the kv-asset-handler package and are having problems with migrating to durable objects since the package still assumes global env vars not wrapped in the env object. is there any workaround other than forking the library?

Wait? You have set more than 10k timeouts within one worker? What are you trying to do?

Weren’t you working on kv-asset-handler recently? Do we have a solution for a module syntax version of it? Doesn’t sound too hard?

I need a timeout after every message I receive from the websocket.

mainly I've been wrapping the request in a synthetic event object so far, but this is inside the library so a fork seems necessary

we could create a pr if that's welcome maybe?

Oh I see. And do you have 10k open setTimeouts? Or 10k in total?

one at a time

Oh hell yeah. PRs are very welcome

that's worrying if you're only keeping one setTimeout per connection and you're still running out - I'll hit that limit too with my project

So your web socket is just open for a long time and eventually you hit 10k. Just out of curiosity, what are you doing in the timeout and how long do you set the timeout? Just so I can help come up with alternatives maybe.

And I will drop a question in the team chat to ask about the limit.

I'm not @User but I'm doing

const restartZombieCheck = () => {
    if (activityCheckTid) {
        clearTimeout(activityCheckTid);
    }
    activityCheckTid = setTimeout(() => {
        socket.send(JSON.stringify({ kind: "Ping" }));
        activityCheckTid = setTimeout(() => {
            // Client didn't respond! Declaring dead...
            closeOrErrorHandler(new CloseEvent("close",
                { "wasClean": false, "reason": "Pong not received.", "code": 1006 }
            ));
        }, this.activityWindowMs);
    }, this.activityCheckMs);
}

I am implementing a protocol that requires the server to send out steady ticks if there are no messages. But the tick time must reset with every message. I don't think there is a way to do that other than a timeout or interval. Can be 30 ticks/sec, and dozens of messages per second.

Thanks 🙂

@User is this 10k a per DO limit?