Hi all I am trying to use s3 for checkpointing and i found t Apache Flink #troubleshooting

Hi all. I am trying to use s3 for checkpointing, a...

Jirawech Siwawut

04/12/2023, 4:29 PM

Hi all. I am trying to use s3 for checkpointing, and i found this error when I tried

Copy code

org.apache.flink.runtime.checkpoint.CheckpointException: Failure to finalize checkpoint.
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.finalizeCheckpoint(CheckpointCoordinator.java:1346) ~[flink-runtime-1.15.1.jar:1.15.1]
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1241) ~[flink-runtime-1.15.1.jar:1.15.1]
	at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:1133) ~[flink-runtime-1.15.1.jar:1.15.1]
	at org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$acknowledgeCheckpoint$1(ExecutionGraphHandler.java:89) ~[flink-runtime-1.15.1.jar:1.15.1]
	at org.apache.flink.runtime.scheduler.ExecutionGraphHandler.lambda$processCheckpointCoordinatorMessage$3(ExecutionGraphHandler.java:119) ~[flink-runtime-1.15.1.jar:1.15.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_301]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_301]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified bucket does not exist (Service: Amazon S3; Status Code: 404; Error Code: NoSuchBucket;

Here is my setup

Copy code

flink-conf.yaml
s3.access-key: accces_key
s3.secret-key: secret_key
s3.endpoint: <http://myendpoint.com|myendpoint.com>
---
env.setStateBackend(new FsStateBackend("<s3://mybucket/checkpoint>"));

I found that it is still connecting to default s3 endpoint from debug log

piby 180

04/12/2023, 5:28 PM

For me, checkpoints on S3 worked with

presto.s3.endpoint : <http://s3.eu-central-1.amazonaws.com|s3.eu-central-1.amazonaws.com>

Jirawech Siwawut

04/12/2023, 5:51 PM

I tried to change my endpoint to

presto.s3.endpoint

no luck. is it possible that it is fixed to s3.amazonaws.com. I found this link

piby 180

04/12/2023, 6:40 PM

not sure.Maybe someone else has more insight on this. For me, I was getting Access Denied Error with the correct S3 keys and correct bucket name. The problem is resolved as soon as I added s3 endpoint. The code was probably looking for AWS_REGION.

👀 1

Jirawech Siwawut

04/13/2023, 1:25 AM

I see. I am trying to connector other endpoint than s3. link. It seems did not work at all 🥲

Jirawech Siwawut

04/13/2023, 10:38 AM

@piby 180 Could you share how do you set up the code and configuration? I checked object in debug mode after i configure this

Copy code

presto.s3.access-key: access
presto.s3.secret-key: secret
presto.s3.endpoint : <http://s3.eu-central-1.amazonaws.com|s3.eu-central-1.amazonaws.com>

I still see that endpoint is not overriden.

Open in Slack

Previous Next