I'm using similar settings, except we have smaller nodes so Xmx == 12g. Things improved a lot when disabling the offheap segment settings:

pinot.server.instance.realtime.alloc.offheap.direct