2023-03-27 171232,799 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Decline checkpoint 145808 by task f05374c958a69e3e10c941a67bc989a4 of job 315916e1be6a757f393973f31df8fc1c at container_e80_1670459712027_0027_01_000002 @ worker-06.hadoop.xtadmins.com (dataPort=37846). org.apache.flink.util.SerializedThrowable: Asynchronous task checkpoint failed. at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:279) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:175) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202] Caused by: org.apache.flink.util.SerializedThrowable: Could not materialize checkpoint 145808 for operator Join(joinType=[InnerJoin], where=[(beneficiary_account_id = beneficiary_account_id0)], select=[created_time, status, beneficiary_account_id, withdraw_request_id, withdraw_amount, withdraw_currency, fee_amount, firm_id, beneficiary_account_id0, encrypted_beneficiary_account_no, swift_code], leftInputSpec=[NoUniqueKey], rightInputSpec=[NoUniqueKey]) -> Calc(select=[created_time, status, withdraw_request_id, withdraw_amount, withdraw_currency, fee_amount, firm_id, encrypted_beneficiary_account_no, swift_code, SUBSTR(CAST(created_time), 1, 10) AS $f59]) (1/1)#2. at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:257) ~[flink-dist_2.11-1.13.6.jar:1.13.6] ... 4 more Caused by: org.apache.flink.util.SerializedThrowable: java.io.IOException: Could not flush to file and close the file system output stream to hdfs:/user/yarn/checkpoints/e5f40fb2170c4fff9461f5eacea40ca9/315916e1be6a757f393973f31df8fc1c/shared/49480ebe-c5fa-459d-a65f-557ce902ff69 in order to obtain the stream state handle at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_202] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_202] at org.apache.flink.runtime.concurrent.FutureUtils.runIfNotDoneAndGet(FutureUtils.java:636) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.streaming.api.operators.OperatorSnapshotFinalizer.<init>(OperatorSnapshotFinalizer.java:54) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.run(AsyncCheckpointRunnable.java:128) ~[flink-dist_2.11-1.13.6.jar:1.13.6] ... 3 more Caused by: org.apache.flink.util.SerializedThrowable: Could not flush to file and close the file system output stream to hdfs:/user/yarn/checkpoints/e5f40fb2170c4fff9461f5eacea40ca9/315916e1be6a757f393973f31df8fc1c/shared/49480ebe-c5fa-459d-a65f-557ce902ff69 in order to obtain the stream state handle at org.apache.flink.runtime.state.filesystem.FsCheckpointStreamFactory$FsCheckpointStateOutputStream.closeAndGetHandle(FsCheckpointStreamFactory.java:373) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.contrib.streaming.state.RocksDBStateUploader.uploadLocalFileToCheckpointFs(RocksDBStateUploader.java:143) ~[flink-statebackend-rocksdb_2.11-1.13.3.jar:1.13.3] at org.apache.flink.contrib.streaming.state.RocksDBStateUploader.lambda$createUploadFutures$0(RocksDBStateUploader.java:101) ~[flink-statebackend-rocksdb_2.11-1.13.3.jar:1.13.3] at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:32) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_202] ... 3 more Caused by: org.apache.flink.util.SerializedThrowable: java.lang.OutOfMemoryError: unable to create new native thread at org.apache.hadoop.hdfs.ExceptionLastSeen.set(ExceptionLastSeen.java:45) ~[dlink-client-hadoop-XT-0.6.1-SNAPSHOT.jar:?] at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:829) ~[dlink-client-hadoop-XT-0.6.1-SNAPSHOT.jar:?] Caused by: org.apache.flink.util.SerializedThrowable: unable to create new native thread at java.lang.Thread.start0(Native Method) ~[?:1.8.0_202] at java.lang.Thread.start(Thread.java:717) ~[?:1.8.0_202] at org.apache.hadoop.hdfs.DataStreamer.initDataStreaming(DataStreamer.java:633) ~[dlink-client-hadoop-XT-0.6.1-SNAPSHOT.jar:?] at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:717) ~[dlink-client-hadoop-XT-0.6.1-SNAPSHOT.jar:?] 2023-03-27 171232,807 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Trying to recover from a global failure. org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold. at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:98) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:84) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1931) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:991) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$declineCheckpoint$2(ExecutionGraphHandler.java:103) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_202] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[?:1.8.0_202] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_202] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_202] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_202] But the environment of my computer is normal, it has this error