Hi Team, We're using flink kubernetes operator fo...
# troubleshooting
j
Hi Team, We're using flink kubernetes operator for flink deployment. We're seeing operator deploying multiple jobs for same job in a cluster. Job-a is being deployed 3 times and we have increased the value of kubernetes.operator.observer.progress-check.interval: 15s. But still it is happening, how can we fix? https://stackoverflow.com/questions/78332390/kubernetes-flinksessionjob-launching-multiple-instance-of-same-job
n
what does it show when you do
kubectl get flinksessionjob
?
I just recently upgraded to the same version of Java and Flink btw
j
Copy code
NAME               JOB STATUS    LIFECYCLE STATE
all-click-to-fix   CANCELED      STABLE
aws-batch                        UPGRADING
aws-bql-batch      CANCELED      STABLE
azure-batch        RECONCILING   DEPLOYED
azure-bql-batch                  UPGRADING
nx-batch                         UPGRADING
nx-bql-batch       RUNNING       STABLE
I see multiple jobs for jobs with state: UPGRADING
And it stays there
Copy code
Reconciliation Timestamp:  1716884457534
    State:                     UPGRADING
Events:
  Type     Reason               Age                   From  Message
  ----     ------               ----                  ----  -------
  Normal   Submit               7m44s (x13 over 81m)  Job   Starting deployment
  Warning  SessionJobException  7m23s (x7 over 81m)   Job   java.util.concurrent.TimeoutException
Copy code
Error:  {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"org.apache.flink.util.FlinkRuntimeException: java.util.concurrent.TimeoutException","additionalMetadata":{},"throwableList":[{"type":"org.apache.flink.util.FlinkRuntimeException","message":"java.util.concurrent.TimeoutException","additionalMetadata":{}},{"type":"java.util.concurrent.TimeoutException","additionalMetadata":{}}]}
  Job Status:
    Job Id:  14aebf0400c164cb1b9a17c9e99bdd73
    Savepoint Info:
n
There is only one of each job though?
Copy code
all-click-to-fix   CANCELED      STABLE
aws-batch                        UPGRADING
aws-bql-batch      CANCELED      STABLE
azure-batch        RECONCILING   DEPLOYED
azure-bql-batch                  UPGRADING
nx-batch                         UPGRADING
nx-bql-batch       RUNNING       STABLE
Regarding these timeouts, I see something similar due to the JAR downloads taking a bit long, but it eventually succeeds after a few timeouts (the JAR is also cached for some time so sometimes it doesn't fail when deploying). Can you have a look at the Flink Operator logs and the Job Manager logs?
j
I see multiple jobs in flink console for job with state: UPGRADING
n
oh you aren't sure why there are multiple session jobs being deployed at all. Okay I see what you mean now. What does your helm chart look like?
j
Why does a job stuck at UPGRADING? and how to RECONCILE again
n
Why does a job stuck at UPGRADING? and how to RECONCILE again
something to try would be to do a
kubectl edit flinksessionjob aws-batch
and change anything in the spec that doesn't matter to you, then have a look at the Job Manager and Operator logs
j
I see job is trying to reconcile and stuck at UPGRADING which is resulting in triggering multiple jobs
How to push a job from UPGRADING state to other state
n
I see job is trying to reconcile and stuck at UPGRADING which is resulting in triggering multiple jobs
these jobs all have different names which would imply they have been submitted independently.
How to push a job from UPGRADING state to other state
as I said, you could either edit the job with
kubectl
, or you can even delete the session job and try and deploy again I can't really say much more without looking at the Helm chart that produces these however
j
Do you wan to take a look at sessionjob manifest?
n
that might help yes
j
Copy code
apiVersion: flink.apache.org/v1beta1
kind: FlinkSessionJob
metadata:
  name: nx-batch
  namespace: dev1
spec:
  deploymentName: flink-cluster
  restartNonce: 124
  job:
    jarURI: https://s3.xyz/az.jar
    parallelism: 2
    upgradeMode: savepoint
    entryClass: com.worker.Worker
    allowNonRestoredState: true
    savepointTriggerNonce: 2
    args:
      - "--jobType"
      - "batch"
      - "--cloudProvider"
      - "nx"
      - "--buildId"
      - "dev1-test"
      - "--operatorParallelism"
      - '{"default":1}'
n
so for me the mystery is that, if this job's name is
nx-batch
, where are those other names coming from? It sounds like you have started recently with this so I'm assuming it's just in dev. As bad as it sounds I would consider deleting the whole flinkdeployment and deploying again.
j
I shared only one job manifest
which is stuck at UPGRADING
n
I shared only one job manifest
I was trying to understand then what you meant by this:
We're seeing operator deploying multiple jobs for same job in a cluster. Job-a is being deployed 3 times
But in any case, whether that's an issue or not, if the job is stuck I would edit one of those
args
in the manifest and monitor the Job Manager and Operator logs
What happened in my case: My Flink Operator didn't have permission to access the JAR I was trying to download. After allowing that my job deployed fine
👍 1
j
We've same permission for other jobs
After job manager restart: All jobs are stuck at UPGRADING
Copy code
NAME               JOB STATUS   LIFECYCLE STATE
all-click-to-fix                UPGRADING
aws-batch                       UPGRADING
aws-bql-batch                   UPGRADING
azure-batch                     UPGRADING
azure-bql-batch                 UPGRADING
nx-batch                        UPGRADING
nx-bql-batch                    UPGRADING
This is weird, we've 35 jobs running on 6 TM nodes for 7 submitted flinksessionjobs
n
This is weird, we've 35 jobs running on 6 TM nodes for 7 submitted flinksessionjobs
in the UI, which jobs are listed? Yes this is weird
j
In UI, there are multiple jobs for each
n
all with the same name? 🤔
j
yes
n
did you check the logs?
j
Copy code
pointDuration=376 ms, finalizationTime=293 ms).
[2024-05-28T09:20:54.706Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 2 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888054696 for job dd9909f36f545573cd61e95ca6f92974.
[2024-05-28T09:20:54.937Z] INFO jobmanager-io-thread-2 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/dd9909f36f545573cd61e95ca6f92974/chk-2/_metadata with MPU ID jQ1n0aJ2cCaC1cjpJcO440TFlvE0x5iT67A0Ur2AqFI3bmOSiauuq3wRi9MZhJhnXwPky9GviqxRfqJY6o8Ie.E03pkk1iu_.Ta9aZc_S9zre5MNEoqB62Aro9_oqJ_YQGjdkfYSInhJYWsF.Sz.U_HZZnp0lP_EJprObg9AVGg-
[2024-05-28T09:20:55.274Z] INFO jobmanager-io-thread-2 CheckpointCoordinator:1335 Completed checkpoint 2 for job dd9909f36f545573cd61e95ca6f92974 (474 bytes, checkpointDuration=374 ms, finalizationTime=204 ms).
[2024-05-28T09:21:20.585Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 12 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888080575 for job 0106acf34013cfd3fb5e3f05fa081db0.
[2024-05-28T09:21:20.884Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/0106acf34013cfd3fb5e3f05fa081db0/chk-12/_metadata with MPU ID nB_8LWtu.6jBmM_uM0Gj3RU4d0X3GBUgG634Q8aDJ3_4m8BmjuM9iomwZoXA7Ob_5GsheeYxIVbQLkLobX_QqHiHoT0wbYfPdj6bEtkmBYcak42xViQXqb0E6upsO3noJI5tDtbK2XlP6KgXIwozedM6JHaOHfne_s5GldXGbck-
[2024-05-28T09:21:21.274Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 12 for job 0106acf34013cfd3fb5e3f05fa081db0 (151512 bytes, checkpointDuration=422 ms, finalizationTime=277 ms).
[2024-05-28T09:21:28.752Z] INFO jobmanager-io-thread-2 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/4001b9cc31b0a0cbce345af64c1232db/chk-1/_metadata with MPU ID A8iTNuozavdeRU5t2IghewbReRMtKNfTPKsYethlZG6WIC0oeoaR2Ht2QQ4WwTUMu9PznH3EPfTYGhRh5qeKTsGBih_GVFai3Uz4b0SUlPudGdHyBohs6tq7n8Dv32zG6mKP8hwjJopyu3P2N1scneF8Q1fkjs0hNKycXiIMiYY-
[2024-05-28T09:21:29.025Z] INFO jobmanager-io-thread-2 CheckpointCoordinator:1335 Completed checkpoint 1 for job 4001b9cc31b0a0cbce345af64c1232db (704602 bytes, checkpointDuration=706348 ms, finalizationTime=159 ms).
[2024-05-28T09:22:33.600Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 2 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888153591 for job 189c02cfd30ddfbde31fbc9dbd43af0f.
[2024-05-28T09:22:34.055Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/189c02cfd30ddfbde31fbc9dbd43af0f/chk-2/_metadata with MPU ID a1_FmepKzt0DIEfRQ1NN8BSjMN3oxuzt9ltfVihJCGV5Vg2mG6BzO5KPWADXpVnl9.zZVUP6eH8n_LFF27v6b6a.KQnFGz_07mojbvCP2Ozlo8JC0DM_tA0kWQinVdxp8wzLE3118p8OUlmpLLL.8krgcnoCAeVGlkpHU8nbETg-
[2024-05-28T09:22:34.476Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 2 for job 189c02cfd30ddfbde31fbc9dbd43af0f (151512 bytes, checkpointDuration=607 ms, finalizationTime=278 ms).
[2024-05-28T09:22:44.103Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 3 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888164094 for job fcb08664619c3a68be8c390640c91c63.
[2024-05-28T09:22:44.327Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/fcb08664619c3a68be8c390640c91c63/chk-3/_metadata with MPU ID lHvNyNtO8Nk4giFSPWkoWGbhjUQLLPdLexmAAtJ1ntazZ_CTr8wadbZXGLd_NAnonFxwdXCx5JaNYxI62jIBtx30pIkttlw3TnBoJh8_ru5Fp5SRcMBBA_JLxTu1TWnvshtIifdL8jxmb8L73XjxCEbl6DbKxS3VhqzCnLtJ1Gc-
[2024-05-28T09:22:44.672Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 3 for job fcb08664619c3a68be8c390640c91c63 (474 bytes, checkpointDuration=365 ms, finalizationTime=213 ms).
[2024-05-28T09:23:01.022Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 12 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888181012 for job 9ccb150d66397272612f286c8296c26a.
[2024-05-28T09:23:01.261Z] INFO jobmanager-io-thread-2 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/9ccb150d66397272612f286c8296c26a/chk-12/_metadata with MPU ID EcpK_6oU5uwKAnYTB_8SbM6oCIH8qiQmfm1ak2X_hOGs9Kdghv7fsQDvPFaUbPb4DiJYrdsxymgL4h_UCQwugYWg9AIjZriP8AbGbqkUVv_OYF9IvDs_Af0jRTKsx5JNxL3x2UwRHJk0dO_tiNr4SmpeI3YTL6EhwnkNwaJnpuE-
[2024-05-28T09:23:01.617Z] INFO jobmanager-io-thread-2 CheckpointCoordinator:1335 Completed checkpoint 12 for job 9ccb150d66397272612f286c8296c26a (474 bytes, checkpointDuration=377 ms, finalizationTime=228 ms).


[2024-05-28T09:23:11.755Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 2 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888191747 for job f588b97cbea2b8e49068e3e8ca4bf5b9.
[2024-05-28T09:23:12.028Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/f588b97cbea2b8e49068e3e8ca4bf5b9/chk-2/_metadata with MPU ID it92Eq4PxkepMevHqcUo7ICXQNL7Y_5PTiedVbc2rF3ql95dHkj0otFh0Ww1bqB2nBM8rARENJCjvif2TTNF41wiME_npPnICH7.wvDOa7hraCbW5tWpVVDbpBe4LmTQSyUgfQTgbCP.5whhri_fQ5IPXtuaqXiiSKoiKx.MIjE-
[2024-05-28T09:23:12.406Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 2 for job f588b97cbea2b8e49068e3e8ca4bf5b9 (117948 bytes, checkpointDuration=395 ms, finalizationTime=264 ms).
[2024-05-28T09:23:15.736Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 2 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888195726 for job 82b591a9b69296e1098f5a53a3f6eeda.
[2024-05-28T09:23:16.019Z] INFO jobmanager-io-thread-2 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/82b591a9b69296e1098f5a53a3f6eeda/chk-2/_metadata with MPU ID 4QIJNDvMIewnBq7p2DZbH5nYs_64n4AWDAID2vYHoIatLTFJ_CFMIhtGRITPMaRgeloq3Zc3PnSGa9Cp0Z4wVyYrjcz.lwoQ.Hf4vNilOCbq80pkrpdPulj57xQ5nQwNGkHvnXtqVrwn204tr_SczWBPEpntzxlpmFSFq8vYbVE-
[2024-05-28T09:23:16.424Z] INFO jobmanager-io-thread-2 CheckpointCoordinator:1335 Completed checkpoint 2 for job 82b591a9b69296e1098f5a53a3f6eeda (151512 bytes, checkpointDuration=415 ms, finalizationTime=283 ms).
[2024-05-28T09:23:29.800Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 3 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888209790 for job 78ad08c310f030570b8f5c42910c7b10.
[2024-05-28T09:23:30.075Z] INFO jobmanager-io-thread-2 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/78ad08c310f030570b8f5c42910c7b10/chk-3/_metadata with MPU ID F_VeenVFr298a9az5Mak8bUmfALV_245NiHBVgk8ZAN_bz1Tgi610b93kORdoWpBGVFoTmJHMNwP3nXje2HSH4v.GBalQlGphqJKFyUisDSc91oGwQIxTB2ifM9j5x8BMII4ZFJ01UOIJYvJ6FEuigUT32BfwdI0oc3_EKweBQE-
[2024-05-28T09:23:30.479Z] INFO jobmanager-io-thread-2 CheckpointCoordinator:1335 Completed checkpoint 3 for job 78ad08c310f030570b8f5c42910c7b10 (151512 bytes, checkpointDuration=410 ms, finalizationTime=279 ms).




[2024-05-28T09:24:46.219Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 13 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888286204 for job 3472663264dc88126da88b902059a1be.
[2024-05-28T09:24:46.652Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/3472663264dc88126da88b902059a1be/chk-13/_metadata with MPU ID z597auRBEWg2ny3EVVLcEXm16yl_Juw5c8pk9p7fSo0jfQX2lt3eH2pPrCODNn9wEHgNrO3Ta3RJ3wQhkCLNoAK1RrkyNbiZfwjdAPtzxozc63qLZhsKn4Zp84zI_IU2NwM_o9a37Uy2cUAO9RB4_JMPYT6FzmfLxkW_sFeFX3g-
[2024-05-28T09:24:47.126Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 13 for job 3472663264dc88126da88b902059a1be (117948 bytes, checkpointDuration=583 ms, finalizationTime=339 ms).
[2024-05-28T09:25:17.845Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 12 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888317836 for job 894f19243303ce2db79159ffaec6aadf.
[2024-05-28T09:25:18.158Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/894f19243303ce2db79159ffaec6aadf/chk-12/_metadata with MPU ID jfg67wzhF1u9IB1GFO8MjcrbbJa5noXwaJj9rcSsJARCH0Ot7dOiD5wv_nJV4QDrcHFFK8GghaqOrYCbpoWDO6wUNinUDIF4n0NS6M.Mc0NJ9dwwxPGNJruj4i6OMLl9qMraCBNkxxQWTCw4IzbgFens0EW.JtbT0PcpSoc.1h4-
[2024-05-28T09:25:18.565Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 12 for job 894f19243303ce2db79159ffaec6aadf (117948 bytes, checkpointDuration=441 ms, finalizationTime=287 ms).
[2024-05-28T09:25:19.160Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 7 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888319151 for job 808c24db23feb1563adc78a627375aba.
[2024-05-28T09:25:19.422Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/808c24db23feb1563adc78a627375aba/chk-7/_metadata with MPU ID xaVKunRDhbZjuBlGt12jJ91GWvNnwsnao4dCqJr_D.iDqZKop1y8dfaBBlqkd_nYYijRTH0xywXHPXMg8nlhQNxOh3CC3.Qz7ArVFJR7CDZ_xLGJpXNEYmiPmD0zoq._pRtsdQluG13Msek5Mbqs57N.wBUgwkmqg2mgQ4HV25s-
[2024-05-28T09:25:19.749Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 7 for job 808c24db23feb1563adc78a627375aba (117948 bytes, checkpointDuration=404 ms, finalizationTime=194 ms).
[2024-05-28T09:25:36.352Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 3 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888336344 for job 30aa2b9048f39b544126feb94385034b.
[2024-05-28T09:25:36.410Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 3 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888336348 for job f0f72bb64a3676523030d2760204a09e.
[2024-05-28T09:25:36.646Z] INFO jobmanager-io-thread-2 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/30aa2b9048f39b544126feb94385034b/chk-3/_metadata with MPU ID Z50wbwhPH.UWpS6yVm.pTt4VyWu85uXJz_fhVwkHBBrfJMRZgEhH7jHiCAB__YxPRK4nA7zjqLqioswKnJRQ93JN7qWPq.b1N47IAGkinNkA.xZl7vfgPzb9otRSO6o4DOzk98biZ47.8xuHrsNTktw8yKrgJvJ5PISoBdabGek-
[2024-05-28T09:25:36.670Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/f0f72bb64a3676523030d2760204a09e/chk-3/_metadata with MPU ID 7ZyoZZRRbPvzVhfwH9jWv.1eW79aAaQXizNjzPO1UcUj_lObGEKJGukvwiBfWKEzHaZ1nLd8kaZDOGUmvpXaO9QSxNt1yq6wSNDLxThqlICpAkDQnvNbTBjUYp18Kjc0G9O_sMXdb6C.OyQw9HuQipOnPUH_ezzeieSKv8iysO4-
[2024-05-28T09:25:37.032Z] INFO jobmanager-io-thread-2 CheckpointCoordinator:1335 Completed checkpoint 3 for job 30aa2b9048f39b544126feb94385034b (117948 bytes, checkpointDuration=439 ms, finalizationTime=249 ms).
[2024-05-28T09:25:37.216Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 3 for job f0f72bb64a3676523030d2760204a09e (117948 bytes, checkpointDuration=434 ms, finalizationTime=434 ms).
[2024-05-28T09:25:50.860Z] INFO Checkpoint Timer CheckpointCoordinator:845 Triggering checkpoint 4 (type=CheckpointType{name='Checkpoint', sharingFilesStrategy=FORWARD_BACKWARD}) @ 1716888350846 for job 5fadad5b1db2444cd91c1a34f26effa1.
[2024-05-28T09:25:51.182Z] INFO jobmanager-io-thread-1 S3Committer:67 Committing fsc-stage-k8s-dev1-cp/5fadad5b1db2444cd91c1a34f26effa1/chk-4/_metadata with MPU ID Vnat4KYCViIq6exdecE0DEkeJuheInqOw1bpnOCbMwiz9nDV9HjbBNlFHA8iVRoUCzmz_zkXVq0YOu6vqJzp9KlzqqfvCp.AthGYaR6L0rwG.kj9X7oVfG18_ufQ6wwzccWnPIjFPQ1TMYhJCbTDT2H0jNqLk5K676AR7Dc1y1w-
[2024-05-28T09:25:51.587Z] INFO jobmanager-io-thread-1 CheckpointCoordinator:1335 Completed checkpoint 4 for job 5fadad5b1db2444cd91c1a34f26effa1 (151512 bytes, checkpointDuration=468 ms, finalizationTime=273 ms).
jm_latest logs
Jobs count increasing for every restart of JM pod, now it's running 41 jobs
Can someone check this issue
n
I don't think I can help without spending quite a bit of time on this. It's very difficult to debug this kind of thing without reproducing it
j
Okay
Can someone help here
g
Which operator version are you using? If not the latest then a good guess is to upgrade to 1.8.0
j
We're using latest one
Copy code
image:
  repository: apache/flink-kubernetes-operator
  pullPolicy: IfNotPresent
  tag: latest
a
@Jashwanth S J Hit the curl request to JM and list the jobs. You might need to manually hit the curl to delete those jobs.
j
Why is it even creating duplicate multiple jobs at first place?
a
If you remove finalizer in sessionjob CR, the sessionjob may not enforce the deletion of the job in flink before it is deleted from the cluster. At least that’s what happened in my case.
j
We didn't remove finalizer