Johannes Cox
08/14/2023, 3:00 PMexecute(). The Job is deployed with the Flink-Kubernetes-Operator in Application Mode without High-Availability enabled. According to the docs for the Application Mode this should be possible. However, instead of deploying multiple jobs, the first job is just being recovered for each new execute(). When I run the job locally in my IDE, this issue doesn’t occur. Does anyone has an idea what is wrong with my setup?
I tested the behaviour with the following job:
import org.apache.flink.api.common.RuntimeExecutionMode;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class MultipleExecutes {
private static final Logger LOG = LoggerFactory.getLogger(MultipleExecutes.class);
public static void main(String[] args) throws Exception {
for (int i = 0; i < 5; i++) {
<http://LOG.info|LOG.info>("Start run " + i);
StreamExecutionEnvironment sEnv = StreamExecutionEnvironment.getExecutionEnvironment();
sEnv.setRuntimeMode(RuntimeExecutionMode.BATCH);
sEnv.fromElements("hello", "world", "apache", "flink").print();
sEnv.execute();
}
}
}
Job Log:
2023-08-14 14:26:03,737 INFO MultipleExecutes [] - Start run 1
2023-08-14 14:26:03,740 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job f7059a41302582a4dbcfcc349fca660b was recovered successfully.
2023-08-14 14:26:03,740 INFO MultipleExecutes [] - Start run 2
2023-08-14 14:26:03,742 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job f7059a41302582a4dbcfcc349fca660b was recovered successfully.
2023-08-14 14:26:03,742 INFO MultipleExecutes [] - Start run 3
2023-08-14 14:26:03,744 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job f7059a41302582a4dbcfcc349fca660b was recovered successfully.
2023-08-14 14:26:03,744 INFO MultipleExecutes [] - Start run 4
2023-08-14 14:26:03,759 INFO org.apache.flink.client.deployment.application.executors.EmbeddedExecutor [] - Job f7059a41302582a4dbcfcc349fca660b was recovered successfully.
This my deployment yaml:
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
name: test-multiple-executes
spec:
image: ...
imagePullPolicy: Always
flinkVersion: v1_17
flinkConfiguration:
taskmanager.numberOfTaskSlots: "1"
serviceAccount: flink
jobManager:
resource:
memory: "2048m"
cpu: 1
taskManager:
resource:
memory: "3072m"
cpu: 1
job:
jarURI: ...
entryClass: ...
parallelism: 4