Hello!, I was testing the flink kubernetes operato...
# troubleshooting
f
Hello!, I was testing the flink kubernetes operator to validate the HA capabilites in a flinkDeployment and I have a doubt, I am adding this to the flinkConfiguration:
Copy code
s3.endpoint: http://my-minio.com:9000
s3.path.style.access: "true"
high-availability.type: kubernetes
high-availability.storageDir: "s3://flink-deployments/ha"
state.savepoints.dir: "s3://flink-deployments/savepoints"
state.checkpoints.dir: "s3://flink-deployments/checkpoints"
So after upgrade the service, restart it using restartNonce flag, or killing the job manager or task manager pods the service recognizes the last jobId and continues working. But, the only problem that I found is when I am recreating the flinkDeployment, in this case it creates a new jobId and ignores the previous checkpoints or ha metadata stored in s3, why happens that? is this an expected behaviour in case of deletion and recreation of flinkDeployments? I verified that the kubernetes.cluster-id is exactly the same before the deletion. I would like to guarantee that in case of redeployments, the same jobId should be re attached to the new flinkDeployment instance
p
When you are deploying a new job, there is a config named
Copy code
flink-deployment:
  ...
  job:
    initialSavepointPath: ...
Where you can define from which savepoint/checkpoint your job should start from. When you use the k8s operator, it should make the trick automatically for you
f
Hello @Pedro Mázala thanks for the answer, when the flinkdeployment is created for the first and there is no previous checkpoint, what initial value can be set to the initialSavepointPath? The goal is to deploy it once, then if a restart o recreation occurs it can use this initialSavePointPath inmediately
p
When it's fresh, this value can be empty. It's not required. So are you using flink k8s operator?
f
Yes, I am using the 1.11
p
So, when you deploy a new job it should have it automatically. What is the value you're using on
job.upgradeMode
?
f
last-state, but when I delete and recreate the flinkDeployment it creates a new checkpoint anyway ignoring the existing one
p
You don't need to delete and recreate. Perhaps this is where the issue is. the operator will make sure to delete and recreate it for you if the job has the same name. This would sort your issue.
f
Correct, the common scenario must be full gitops where the operator manages the lifecycle, but I want to prevent unexpected behavior in case of human or kubernetes errors and recover the right checkpoint always
It is an on premise scenario
p
Gotcha. So to achieve that you will have to fetch the latest checkpoint/savepoint and add it on your deployment file via
job.initialSavepointPath
on each deployment I'm afraid
f
So in a gitops release process we need two steps: 1. empty run that creates the jobId and initial checkpoint 2. update the job.initialSavepointPath in order to use this checkpoint always. Quite tricky but works as expected 🙂
p
Nice! If you fully rely on the rolling mechanism this would be smoother. But I understand the need of a closer grip