Hi team , we have a Flink app which is reading fro...
# troubleshooting
s
Hi team , we have a Flink app which is reading from s3 and publishing to Kafka topic. It seems like after its done reading and publishing, a few hours later or so, its automatically redeploying and reading/publishing the data again(approx once per day). I have checked the gitlab ci/cd pipelines to ensure it wasn't triggered manually. I am using Flink’s
FileSource
to read from an S3 bucket. The source is configured to read from a specific prefix (e.g.,
s3://my-bucket/creditAccountOpened/
). It reads all files in that path when the job starts. These are the following parameters i have tried so far to prevent it from redeploying which haven't worked: • "kubernetes.operator.cleanup.terminal-jobmanager.ttl": "300s" • "kubernetes.operator.jm-deployment.shutdown-ttl": "600000" (shuts down the app but immediately reconciles and redeploys) • cleanupPolicy: afterJobFinished: DeleteCluster (not applicable for flink deployment) • ttlSecondsAfterFinished (not applicable for flink deployment) Is there any other way I can ensure that the job is not redeployed on its own once its FINISHED besides deleting the deployment manually? Thanks in advance!
p
why does it restart when finished? Even tho it happens Flink should keep state of things already processed. Are you relying on checkpoiinting?