Any other advice about how to utilize this machine to it's full extent? I suppose I can try running 72, 96, etc cores and see if it shows any improvement over 64

wondering too if there aren't any ways to optimize the decomposition method, the numanodes or anything else

i haven't spent much effort attempting, but did you try

--bind-to numa

?

Was just looking at the 4th gen release dates, seems like those processors started coming out in Nov 2022, and we got the final bid on this machine around Dec 2023. We were requesting they optimize it for OpenFOAM.... we should have included some better langauge about benchmarking.

I tried something similar, but I'll try that too, thanks

yep, even if they were limited to 3rd gen, the cheaper 7573X processors would have been a better bet

you can also try turning off SMT but in my experience it didn't change much and made other applications slower

We have an overset simulation that runs perfectly fine on a local machine in parallel with 6 CPUs. However, on the cluster with more CPUs (e.g., 64), it runs fine for the first few time steps but then gets stuck. Slurm reports that the job is still running, but the simulation output stops. I checked the output before the last line and found nothing suspicious. I also tried activating multiple debug flags, but to no avail. I tried to increase the RAM for each CPU

--mem-per-cpu=3GB

but that doesn't help. Any ideas on how to tackle this (without compiling OpenFOAM in debug mode, etc.)?

Write after every step to get fields before the forever-loop, see whats wrong in there?

How could it be caught in a forever-loop.. iterations have a max limit typically..?

the output is similar to this post: https://www.cfd-online.com/Forums/openfoam-community-contributions/228713-solved-wavedymfoam-stuck-after-calculating-motion-body.html

I have these debug switches on the system/controlDict:

DebugSwitches
{

  sixDoFRigidBodyMotionConstraint 3;
  overset 3;
  lduMatrix 3;
  SolverPerformance 3;
  inverseDistance       3;
  inverseFaceDistance 3;
  inversePointDistance 3;
  inverseVolume 3;
  vanLeer 3;


}

@tkeskita : FWIW the case isn't mine, so I have a limited access to it. So far I recommended to activate those flags. But to find the appropriate one (if any) is tricky

I would like to know about a tool/utility to see what's happening exactly on the cluster.

But I agree with your recommendation

It seems that there is some infinite loop involved here

If you can't ssh to the compute node, then maybe some kind of post-mortem analysis. Maybe limit job run time in Slurm and try to catch error messages about what is was doing when it was killed? 🤔

It's is stuck at this output (copied from the link above because it is similar):

6-DoF rigid body motion
    Centre of rotation: (0 0 -0.000748691)
    Centre of mass: (0 0 -0.000748691)
    Orientation: (1 0 0 0 1 0 0 0 1)
    Linear velocity: (0 0 -0.0381259)
    Angular velocity: (0 0 0)

No I mean if it would print some kind of stack trace.. but maybe it won't

the stderr is empty

No help then. Maybe try another decomposition method or number of processors? This is going to be like stabbing in the dark though...

Yeah, Let's see. I will report any progress in future

On the cluster, when the simulation is run with only 6 CPUs with

--mem-per-cpu=5GB

it works just fine, like the local machine.

i like linux simply because of better multiuser support

I ran a simulation of wind turbine on hoc for different number of processors and the results i am getting is different for every selected number of processors. Any idea why this error is occurring?

@silentspirit , what is the magnitude of this difference? You will get different results as you are changing the underlining linear equations of your system as you change the number of decompositions.

Almost 30-40% difference in values

Is the geometry fairly complex? I have seen simulations that run stable on one core configuration but explode when using a different one.

I have some queries about how to install openfoam in a cluster that is built using OpenHPC

Can anyone guide me to some documents that I can refer to?