Hello everyone, I have a few questions regarding resources for jobs that contain both CPU and GPU based workloads.
I have a job which is made up of three tasks that serve the purpose of preparing and gathering data for the fourth task which runs model inference using the data mentioned as input.
I use DJL to run my own pytorch model. I'm running on a native Kubernetes cluster where each taskmanager pod has 8 cpus, 25gb of ram and 1 GPU.
The way I ran the cluster initially was with 1 task slot per taskmanager and parallelism of 15 since I have 15 pods meaning 15 GPUs and only wanted one Inference subtask to run per card.
My main issue is that I suspect that I might not be getting the most out of my hardware, specifically the cpus. Therefore I have the following questions:
1) Since DJL leverages JNI and native code, does it even conform to the spec of the assigned task slot? Does any native code called from within a flink job?
2) Does Flink support multiple CPU cores per task slot? Most use cases that I've seen have resorted to using more slots when additional cores are available so that there's at most 1 per task slot. When running with the parallelism I've mentioned earlier, upon querying the rest API, I can see that the total resources of my taskmanagers say 1 CPU but the hardware section says 8.
3) Would it be wiser to use fine grained resource management to split the resources into two groups, one for the first three tasks and another for inference? This sounds good since it would allow me to use higher parallelism and leverage more task slots without having the risk of multiple slots from the same taskmanager being assigned to Inference and therefore ruining the load balancing between the cards. If it is the better option, could any of you provide me with even the most basic example? I've had trouble finding an actual working example online and have ran into 'couldn't aquire minimum resources required exceptions'.
Sorry for the lengthy questions.
I would be very grateful if you could help me.
Thanks π