Hello Team, I am hosting Apache Flink 1.19.1 in a...
# troubleshooting
p
Hello Team, I am hosting Apache Flink 1.19.1 in a Kubernetes (EKS) cluster. I have configured
state.checkpoint-storage
to use the filesystem, with checkpoints and savepoints configured to an S3 bucket. While I am able to host Flink, I encounter an error when attempting to submit the new JAR from the dashboard. The error is as follows:
Copy code
2024-07-02 12:08:25,507 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher     [] - Job278686382676784632786 reached terminal state FAILED.
org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
	at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: java.io.FileNotFoundException: Cannot find meta data file '_metadata' in directory 's3://test/savepoints'. Please try to load the checkpoint/savepoint directly from the metadata file instead of the directory.
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
	... 4 more
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: Cannot find meta data file '_metadata' in directory 's3://test/savepoints'. Please try to load the checkpoint/savepoint directly from the metadata file instead of the directory.
	at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
	at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114)
Why I am getting this error for new JAR? Any assistance in resolving this issue would be greatly appreciated.
p
by the error, it looks like you’re trying to restore state from a place which has no state stored. It could be related to missconfiguration on flink-config.yaml
p
Hi @Pedro Mázala Thanks for your response! In my application code, I have configured the state backend and checkpoint directories as follows:
Copy code
// set state backend
configuration.setString("state.backend", "filesystem");
// set the checkpoint directory, which is required by the RocksDB state backend
configuration.setString("state.checkpoints.dir", properties.getProperty("checkpoints.dir"));
configuration.setString("state.checkpoint-storage", "filesystem");
// set the savepoint directory
configuration.setString("state.savepoints.dir", properties.getProperty("savepoints.dir"));
configuration.setString("execution.savepoint.path", properties.getProperty("savepoint.path"));
configuration.setString("execution.savepoint-restore-mode", "CLAIM");
let me know if I have configured something incorrectly?
I need to utilize savepoints and checkpoints for fault tolerance in my job. Checkpoints should be taken at regular intervals, while savepoints should be taken manually whenever needed. These mechanisms should be active only after the job starts running. However, it seems the system is looking for savepoints or checkpoints at the time of job submission, which is unexpected.
s
@Pavan I think when the application tries to create this file at the application start-up. It is not able to do so. The message you are seeing is likely an uber level exception. Is the
test
folder already created in the S3 bucket?
p
@Saurabh Singh This issue is resolved now.
If WE don't want our job to restore from the savepoint or checkpoint metadata at the early stage, We should remove or comment out the configuration for
execution.savepoint.path
. This tells Flink not to restore from any specific savepoint. This was configured on my jar hence it was looking for savepoint
👍 1