livekit server (or dev mode) isn't the reason for the higher latency you are observing. that sounds like it's related with the publishing side.
I would focus on your GStreamer pipeline and see what's happening there.. I have not had a chance to play with the gstreamer plugin that was just released.