https://flink.apache.org/ logo
Join Slack
Powered by
Hello folks, We are using flink 1.4.2 and using A...
# troubleshooting
i
Hello folks, We are using flink 1.4.2 and using AWS S3 as the state backend for storing savepoint and checkpoint , evertything was running fine from couple of months but recently we are encountering one issue where jobmanager restarts the job by giving below error:-
Copy code
AsynchronousException{java.lang.Exception: Could not materialize checkpoint 927 for operator pacing reduce: impression (1/1).}
	at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:948)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Could not materialize checkpoint 927 for operator pacing reduce: impression (1/1).
	... 6 more
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not flush and close the file system output stream to <s3://c6-qa2-s3-uw2-mapr-data/flink/tmp/checkpoints/e719bfc35f191baeaab7ce7aaf87f9b7/chk-927/4818d320-b149-47fe-b99e-510b763edb8a> in order to obtain the stream state handle
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
	at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
	at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
	... 5 more
	Suppressed: java.lang.Exception: Could not properly cancel managed keyed state future.
		at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:91)
		at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.cleanup(StreamTask.java:976)
		at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:939)
		... 5 more
	Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Could not flush and close the file system output stream to <s3://c6-qa2-s3-uw2-mapr-data/flink/tmp/checkpoints/e719bfc35f191baeaab7ce7aaf87f9b7/chk-927/4818d320-b149-47fe-b99e-510b763edb8a> in order to obtain the stream state handle
		at java.util.concurrent.FutureTask.report(FutureTask.java:122)
		at java.util.concurrent.FutureTask.get(FutureTask.java:192)
		at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:43)
		at org.apache.flink.runtime.state.StateUtil.discardStateFuture(StateUtil.java:66)
		at org.apache.flink.streaming.api.operators.OperatorSnapshotResult.cancel(OperatorSnapshotResult.java:89)
		... 7 more
	Caused by: java.io.IOException: Could not flush and close the file system output stream to <s3://c6-qa2-s3-uw2-mapr-data/flink/tmp/checkpoints/e719bfc35f191baeaab7ce7aaf87f9b7/chk-927/4818d320-b149-47fe-b99e-510b763edb8a> in order to obtain the stream state handle
		at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.closeAndGetHandle(FsCheckpointStreamFactory.java:336)
		at org.apache.flink.runtime.state.heap.HeapKeyedStateBackend$1.performOperation(HeapKeyedStateBackend.java:397)
		at org.apache.flink.runtime.state.heap.HeapKeyedStateBackend$1.performOperation(HeapKeyedStateBackend.java:339)
		at org.apache.flink.runtime.io.async.AbstractAsyncCallableWithResources.call(AbstractAsyncCallableWithResources.java:75)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at org.apache.flink.util.FutureUtil.runIfNotDoneAndGet(FutureUtil.java:40)
		at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointRunnable.run(StreamTask.java:894)
		... 5 more
	Caused by: java.io.IOException: org.apache.flink.fs.s3presto.shaded.com.amazonaws.AmazonClientException: Unable to complete transfer: Could not initialize class org.apache.flink.fs.s3presto.shaded.com.amazonaws.util.Base64
		at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem$PrestoS3OutputStream.uploadObject(PrestoS3FileSystem.java:1036)
		at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem$PrestoS3OutputStream.close(PrestoS3FileSystem.java:987)
		at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
		at org.apache.flink.fs.s3presto.shaded.org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
		at org.apache.flink.fs.s3presto.shaded.org.apache.flink.runtime.fs.hdfs.HadoopDataOutputStream.close(HadoopDataOutputStream.java:52)
		at org.apache.flink.core.fs.ClosingFSDataOutputStream.close(ClosingFSDataOutputStream.java:64)
		at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.closeAndGetHandle(FsCheckpointStreamFactory.java:319)
		... 11 more
	Caused by: org.apache.flink.fs.s3presto.shaded.com.amazonaws.AmazonClientException: Unable to complete transfer: Could not initialize class org.apache.flink.fs.s3presto.shaded.com.amazonaws.util.Base64
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.transfer.internal.AbstractTransfer.unwrapExecutionException(AbstractTransfer.java:286)
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.transfer.internal.AbstractTransfer.rethrowExecutionException(AbstractTransfer.java:265)
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.transfer.internal.AbstractTransfer.waitForCompletion(AbstractTransfer.java:103)
		at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem$PrestoS3OutputStream.uploadObject(PrestoS3FileSystem.java:1030)
		... 17 more
	Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.fs.s3presto.shaded.com.amazonaws.util.Base64
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.util.Md5Utils.md5AsBase64(Md5Utils.java:104)
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1620)
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:133)
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:125)
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:143)
		at org.apache.flink.fs.s3presto.shaded.com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:48)
		... 4 more
Caused by: [CIRCULAR REFERENCE: java.io.IOException: Could not flush and close the file system output stream to <s3://c6-qa2-s3-uw2-mapr-data/flink/tmp/checkpoints/e719bfc35f191baeaab7ce7aaf87f9b7/chk-927/4818d320-b149-47fe-b99e-510b763edb8a> in order to obtain the stream state handle]
Can anyone please help here, I am stuck in int from couple of days now
m
I'm hoping you're on a higher version then Flink 1.4.2
i
No we are on 1.4.2
m
Then there's basically no hope 😢 That version is more then 5 years old
i
Yeha I know we are using old version but everthing was working fine and we have not changed anything with the confs
m
It could very well be that another component that you're leveraging (like your object store) has been updated to a newer version that doesn't support the implementation anymore that Flink 1.4 is still using
i
we are using presto jar to interact with aws s3
m
Yes, so if AWS has made a change that requires a specific minimum client version to connect to S3, that could very well be the cause of the problems
i
hmm that might be , can we update presto jar without updating flink version.
m
No you can't
You're honestly in a really bad spot at the moment. You're running a version that isn't supported in the Flink community (only the last two versions are supported) and upgrading will be quite painful, having to go over all the different release notes for each version
i
One more thing our production is running well with the same version and configs , but whenever we are touching any lower env by deploying a jar this issue starts, Can you Please explain why it is running fine in prod.
Though we have decided not to any more deployment in prod till we resolve this issue in lower env's.
m
I can't explain that. That depends completely on your setup and everything, that's impossible to diagnose from the outside
i
Also can you Please suggest any solution to it so that we can move forward.
I can't explain that. That depends completely on your setup and everything, that's impossible to diagnose from the outside ---> Agreed Martijn
m
Please suggest any solution to it so that we can move forward.
You need to start upgrading Flink into a version that's still supported
i
hmm, That is bit challenging for us and flink is very critical for our bssiness.
But Thank you so much for your suggestions and time. @Martijn Visser, Really appreciate that.
m
flink is very critical for our bssiness.
Being the devil's advocate here, but if that was the case, then upgrades wouldn't have been postponed for such a long time πŸ˜…
πŸ˜€ 1
i
Agreed @Martijn Visser
Open in Slack
PreviousNext
Powered by