Theres a fun race with DOs

if a DO calls another DO the msgs can arrive out of order

so need to implement almost a TCP layer ontop of the fetch

with buffers/queues

this happens I think because of the async awaits being not scheduled in order

inside v8/jsruntime itself

the in_progress message got delivered a few milliseconds after the done message, but the in_progress message came first.

there is many DO called User, where the user connect websocket to. another DO pushes update events (fetch) to the USER DO

if the other DO gets 3 events in order, say 1 2 3, then it pushes those events to User DO via fetch (async awaiting the fetch), the User DO may get those fetches out of order

i think this is due to how javascript async/await works

but its a tough problem, i think instead of fetching between the DOs it should be slightly more abstracted to a mailbox (like AKKA persistent mailbox, or erlang in-call-order mailbox)

whats happening I think is when

update_job comes in

awaiting the request body for example, causes it to possibly be scheduled before/after another status update

then depending on when everything arrives and what the VM decides to do, the events could go out of order

the fix isnt pretty but it would probably be batching requests, so there is only 1 inflight request between any 2 DOs. creating a queuelike system

for now il just put a session nonce on my messages and drop any status update messages out of order, maybe timestamps but i heard the DO clocks could be off by a few milliseconds between eachother

like DO1 clock could be 3ms ahead of DO2 clock

depending what hardware it spun up on

I'm not sure I follow, since I don't know the details of what all your code is doing. A reduced code sample showing the issue might help? Generally speaking though, Workers (and Durable Objects) handle requests concurrently, as soon as the event loop is yielded (like with

await

) other requests may be processed up until they, too, yield the event loop.

yea i think it might be storage related too and i need to use transactions

for atomic read/update/write operation

yea the msgs are arriving on the same millisecond lol

any update on how DO will be billed for folks with Workers Bundled?

k it seems due to not using transactions

when requests arrive the way the VM seems to work on cloudflare is it ticks like x interval and pops all the fetches

they go into a await storage.get and both get the same value, then writeback

because there is no way to make the operations on storage sequential

Something interesting would be a compare&update that is atomic on storage to not go into a transaction