Hello everyone :wave: I want to submit multiple b...
# troubleshooting
j
Hello everyone 👋 I want to submit multiple batch jobs by sequentially calling
execute().
The Job is deployed with the Flink-Kubernetes-Operator in Application Mode without High-Availability enabled. According to the docs for the Application Mode this should be possible. However, instead of deploying multiple jobs, the first job is just being recovered for each new
execute()
. When I run the job locally in my IDE, this issue doesn’t occur. Does anyone has an idea what is wrong with my setup? I tested the behaviour with the following job:
Copy code
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class MultipleExecutes {
  private static final Logger LOG = LoggerFactory.getLogger(MultipleExecutes.class);

  public static void main(String[] args) throws Exception {
    for (int i = 0; i < 5; i++) {
      <http://LOG.info|LOG.info>("Start run " + i);

      StreamExecutionEnvironment sEnv = StreamExecutionEnvironment.getExecutionEnvironment();
      sEnv.setRuntimeMode(RuntimeExecutionMode.BATCH);
      sEnv.fromElements("hello", "world", "apache", "flink").print();

      sEnv.execute();
    }
  }
}
Job Log:
Copy code
2023-08-14 14:26:03,737 INFO  MultipleExecutes                                       [] - Start run 1
2023-08-14 14:26:03,740 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job f7059a41302582a4dbcfcc349fca660b was recovered successfully.
2023-08-14 14:26:03,740 INFO  MultipleExecutes                                       [] - Start run 2
2023-08-14 14:26:03,742 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job f7059a41302582a4dbcfcc349fca660b was recovered successfully.
2023-08-14 14:26:03,742 INFO  MultipleExecutes                                       [] - Start run 3
2023-08-14 14:26:03,744 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job f7059a41302582a4dbcfcc349fca660b was recovered successfully.
2023-08-14 14:26:03,744 INFO  MultipleExecutes                                       [] - Start run 4
2023-08-14 14:26:03,759 INFO  org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job f7059a41302582a4dbcfcc349fca660b was recovered successfully.
This my deployment yaml:
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: test-multiple-executes
spec:
  image: ...
  imagePullPolicy: Always
  flinkVersion: v1_17
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "1"
  serviceAccount: flink
  jobManager:
    resource:
      memory: "2048m"
      cpu: 1
  taskManager:
    resource:
      memory: "3072m"
      cpu: 1
  job:
    jarURI: ...
    entryClass: ...
    parallelism: 4