Hey all, I’m trying out the K8s operator and trying to deploy a simple job. For now, I’ve deployed the “basic-session-deployment” (https://github.com/apache/flink-kubernetes-operator/blob/main/examples/basic-session-deployment-only.yaml) and then uploaded the jar file for my job using the “Submit new job” section of the webUI. However, I expected to now see that jar file in the filesystem of the deployment pod but I can’t find it…. My next step was to deploy the job (using the FlinkSessionJob CRD) and reference the location of my jar file using the

jarURI

. So first question, any ideas where I should be looking for my jar file? And second, is this the right way to handle deployments? I’ll be deploying several jobs eventually, and assumed I would want to separate the FlinkDeployment from the actual jobs. (but maybe that’s not the best way?)

The way I did it is building a custom image with the jar to be used by the deployment

Thanks Max, can you explain a little more what you mean? Do you mean that you built a jar which contains both the Flink task manager as well as your application code?

Build a docker image based on

flink

and place your jar inside. e.g.

FROM flink:1.17

COPY /path/to/my/jar /opt/flink/myjar.jar

then you can refer to it in your manifest

...
spec:
  image: my_custom_image:latest
  job:
     jarURI: local:///opt/flink/myjar.jar

Interesting, ok. So if you have multiple FlinkDeployments, does that mean each has its own webUI as well?

I’m still new to Flink, so there might be a better way… but from what I saw, yes

ok. Yeah I’m very new to Flink as well. Maybe this is the difference between “application” and “session” deployments? I clearly have more reading to do. 🙂.

same. If you find an answer please do share 🙂

I assumed you just have a single webUI (and therefore a single cluster of Flink taskmanagers), and then separately you’d define/deploy jobs to that cluster.

That’s what I understand from the quick start guide https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/

@Alex Brekken when you are using the operator you should not be using the WebUI to submit jobs. Any interaction that is supposed to change the Flink job (for example submitting a job 🙂) needs to go through the operator, otherwise it will not be aware and work against you.

@Marton Balassi is my understanding correct and every job should be deployed as a separate

FlinkDeployment

? If so, is there a way to have a unified WebUI for all of them?

Thanks @Marton Balassi that makes sense, but what would be a good practice for how to handle getting my jar file deployed? Can I use my own container registry (dockerhub, ECR, etc) and reference the image that way in the CRD? The examples in github all seem to be using the

jarURI

and referencing a “local” jar file which ships with Flink.

For production jobs we recommending using the application mode where you have a single FlinkDeployment that defines the resources and the job itself. It's basically a single job Flink cluster , isolated from other jobs.

Guys, sorry to be troubling you again, but what is the best practice of having multiple executes in the same Flink app? I see that doing it in a single FlinkDeployment doesn’t work. I know it can work as a statement set, but I do like the separation between the jobs

@Gyula Fóra Since that is recommended for Production, If we have 40 jobs, do we end up with 40 web UI for all those app mode clusters if in case we go with FlinkDeployment with application mode for all 40 jobs?