Quick question. When I'm running the GitHub image classification example I usually use the max_concurrency to match the number of GPUs. For example if I have 8 GPUs I use max_concurrency=8. However in the Ray Cluster overview page in the Usage section I see 8 CPUs and 8 GPUs being used but each gpu worker has many cpus since I'm using like g5.48xlarge instances. I'm seeing 8/1000 cpus being used. Is that just a UI cosmetic issue or do I need to do something else so that the CPU operations like download and decode of the image use more CPUs?

It looks like the ray dashboard has two cpu measurements - logical and hardware (https://docs.ray.io/en/latest/ray-observability/getting-started.html#dash-metrics-view). What does the Node CPU graph look like?

Hi @Chris Kellogg I'm using a g5.24xlarge which has 48 physical cpus and 96 vCPUs. When I use 8 gpus I'm using 2 nodes. So 48 * 2 physical cores and 96 * 2 vCPUs. But in the Ray Dashboard overview I see 8/ (96* 2) CPUs in used and 8/8 GPUs in used since I only set the max_concurrency flag for the daft.cls for model inference. Should I set max concurrency on all the cpu functions?

I will have some more time tomorrow to look into it. When running the GitHub image classification example are you changing anything in the script or just using the settings in the code base? In the ray dashboard "Metrics View" do you see more than 8 cores in the hardware utilization graph (Node CPU)?

@Chris Kellogg I don't have the metrics view setup, I do have graphana, so I check does and report back. What I'm looking at in the dashboard is the image below: which is in the overview tab or the job tab itself. In the example image below I have 4 nodes, 360 CPUs and 16 GPUs. If I run the image benchmark example with gpus=16 I get in the Total Usage 16/360 CPUs and 16/16 GPUs flagged as being used. I'll double check my graphana metrics to see what it shows there but was wondering it the reported usage in the dashboard is a visual bug or actually only using 1 CPU per GPU requested.

Just following up. I think it's mainly a ray ui issue. The cpu usage shown in the Resource Status is related to how many cpus are requested by the ray actors. So in the example the udf actor is requesting 1 cpu (default if not specified in the daft.cls) and there are 8 (max_concurrency). That's why it shows 8.0/<Total CPU>. The swordfish actor does not request any cpus, however it will try to use as many cpus as possible on the host. I ran several test and did see the swordfish actor was utilizing more than what is shown in the Resource Status. In my testing I found that the script runs faster when put on nodes with a single gpu, so 8 instances with 1 gpu has performed the best for me.

@Chris Kellogg thanks for looking into this. Makes sense. You think that the reason why 8 nodes 1 gpu per node performs better is because if we have a 2 node with 4 GPUs each the swordfish will be competing with each other since they will be in the same node?

At this point I don't know. There would need to be some more investigating.