Hi everyone, I am trying to deploy a pyflink appli...
# troubleshooting
r
Hi everyone, I am trying to deploy a pyflink application using Kubernetes operator. My purpose is to have a parallelism of 1, and I want to distribute different subtasks among multiple task managers. So, ultimately something like 7 task managers handling different tasks of a single job with parallelism 1. I tried multiple configuration to achieve this but I end up getting task managers equal to the parallelism which is not my goal. Is it possible to achieve this configuration? If yes, how??
c
I think typically no, you would rather increase the parallelism to spread work across task managers. Is there a reason you need to keep parallelism 1?
r
If there is high parallelism, I will end up having multiple subtasks for a single operator. This means higher parallelism will help if there are multiple keys to distribute. My purpose is to distribute operators so that each subtask gets sufficient resources. Increasing memory configurations for a single task manager does not help because it causes GC isssues.
The issue started with OOM errors. I kept increasing and tuning memory configurations of task manager and ended up allocating large memory size to the task manager. Now I have am facing GC stallls making my task manager unreachable to job manager causing cascading failure.
c
Do you only have 1 key available? I guess that will be problematic since you need all the state to fit in 1 TM I'm not sure if there is a way to force task splitting in the way you want unfortunately
r
Not one but limited keys. And per key there is significant amount of state and processing required.
p
hello, was there a solution for this?