I think we need to come up with a way to really reduce the

GPU vram

usage for cellpose. We have logically partitioned gpus (24gb each) and the cellpose jobs failed on those. I had to request an entire

72gb nvidia_h200_mig_3g.72gb

gpu just to run 1 cellpose job and that consumed around 68.5Gb vram.