@kandelabr @tkeskita @Yann thanks for the tips.
So the issue is solved, I am not 100% sure what exactly, but it's the mesh.
At some point I checkMesh'd the 2 problematic processor and one of them gave segmentation fault directly. However, the neighbour ran correctly, and was not sharing faces near an interface. So I suspect it's just local shitty mesh.
checkMesh -allGeometry -allTopology was complaining about some too small faces/edges. The mesh are from Fluent Meshing, I had limited power there to "increase" smallest faces/edges, but it did the trick. It's luck at this point