I'd highly suggest livekit for it, I didn't use mediasoup, but it seems to be limited on client libraries, and it's based on JS and Rust, I rather have golang based services, so I went to PION first (sfu and ion cluster) and then switched to Livekit, which is way more supported and has more features.
about the server, check out the
benchmarks on livekit . I think you will only run into performance issues if you enable livekit-recorder to record the room sessions as video files, that is a very resource hungry feature, You can also deploy livekit in your server, and run the load-tester on that URL above, to see what kind of results you get.