Age is hell. Time only moves faster

it's also interesting that you need to set the

how

argument to

SHUT_RD

or

SHUT_RDWR

but not

SHUT_WR

for it to cancel the

accept

.

$ zig run accept_client.zig 
info: accepting clients on 7788
warning: shutting down the accepting socket 4 with how value .SHUT_WRITE
info: shutdown: .{ .user_data = 2, .res = 0, .flags = 0 }, cqe.res: 0

This is not something i expected, shouldn't the res field contain an errno here 🤔

Not for jens it seems ....

I've been experimenting a bit with

io_uring_prep_shutdown

to see how it interacts with various other io operations. i set up this tcp server which accepts a client, sends it 1mb, then closes the socket. the client on the other hand, connects, waits for enough milliseconds to allow the socket to fill up with the 1mb of data, then shuts down the socket's read side, and starts issuing `io_uring_prep_recv`s. And during the reaping of CQEs I get these:

info: recv: .{ .user_data = recv, .res = 128000, .flags = 4 }
info: recv: .{ .user_data = recv, .res = 95232, .flags = 4 }
info: recv: .{ .user_data = recv, .res = 411136, .flags = 4 }
info: recv: .{ .user_data = recv, .res = 285696, .flags = 4 }
info: recv: .{ .user_data = recv, .res = 128512, .flags = 4 } // received EXACTLY one MB up to here.
info: recv: .{ .user_data = recv, .res = 0, .flags = 4 }

the flags = 4 is for

IORING_CQE_F_SOCK_NONEMPTY

which i found strange in the last CQE with res = 0 since there is no more data to read but it made me suspect that the server closing the socket is probably what's causing it, and it turns out my intuition is correct, here's what it looks like when the server doesn't close the client socket:

info: recv: .{ .user_data = 2, .res = 128000, .flags = 4 }
info: recv: .{ .user_data = 2, .res = 95232, .flags = 4 }
info: recv: .{ .user_data = 2, .res = 411136, .flags = 4 }
info: recv: .{ .user_data = 2, .res = 285696, .flags = 4 }
info: recv: .{ .user_data = 2, .res = 128512, .flags = 0 }
info: recv: .{ .user_data = 2, .res = 0, .flags = 0 }

can someone confirm whether

IORING_CQE_F_SOCK_NONEMPTY

has some sort of dependence on TCP segments like FIN?

oh wait, I just checked

test/accept-non-empty.c

, just learned that SOCK_NONEMPTY is also used for accepts to signal that there are more pending connections, neat stuff

I have thoughts about io-uring in Tokio. Are you coming to LPC? If so, could we chat about it there?

What are your thoughts?

I think for the majority of user workloads, just dispatching file I/O through io_uring is sufficient

epoll is likely more than enough for most networking workloads common users have

as is probably evident by now, I am not there this year! had some other life items get in the way... but I'd be interested in your thoughts, I've had various folks re-write those parts with great success, but not contribute them upstream afaik

oh, I'd be interested to hear what people did

The current implementation is pretty ... hard-coded. I'd like to allow user-defined io-uring ops with Tokio.

user defined io_uring ops?

as-in, not natively supported by Tokio

when Tokio gets the CQE, execute some random user-provided function

gotcha

that's actually how tigerbeetle does it with its io_uring abstraction

although, i'd love to see how you'd handle chained SQEs (and therefore mutliple CQEs) with callbacks being used

I think you can just have the user data be a pointer to a user data structure storing a function pointer. On CQE you dereference the pointer to read the functoin pointer, then call it with the user data and the CQE.

yeah that's how it goes

but for linked requests, which CQE do you call the callback on

It could be brittle to rely on a function pointer being 8 bytes across all platforms

Oh, I see, I think I misread

user data isn't a function pointer, but a pointer to a function pointer

multiple calls, one for each CQE

Out of curiosity, how does that interplay with the runtime processing CQEs for users normally?

How do you differentiate between something you own as the runtime author vs something someone is trying to manually author?

I keep a slotmap of

enum OpType

and assign each user_data an index into the slotmap, so then from there it's just a normal

match

kind of thing

I was thinking we could just use the same mechanism for Tokio's own ops

is there a way to send multiple (different) udp messages to different destinations ? like with sendmmsg using io_uring and which doesn't involve using multiple SQEs in the process of doing so?