Hi, I've got a stateful Flink job with parallelism...
# troubleshooting
v
Hi, I've got a stateful Flink job with parallelism of 250 that fails at writing checkpoint on s3 (
s3p://
) because of request rate limits throttling. I've checked and there are ~3800 objects in s3, which seems to reach the hard limit of max 3500 puts/sec Is there any way to circumvent this limitation? Look like a blocker to use Flink for big parallelism stateful use cases 😞 To be mentioned that I use RocksDb as state backend
Not sure if that possible (configurable), but it would help if each TM wrote it's part of checkpoint in a s3 "subfolder" because the request rate limits seems to apply on per "folder" basis
p
The checkpoint should be written by the JM (orchestrating it). But in the other hand I'd try using
s3a
protocol and enabling entropy injection to solve this issue.
v
is entropy injection supported in Flink 1.18?
v
well... seems like a promising idea
I'll try it. Thanks a lot for suggestion
p
Let me know if it sorts it out 😁
v
@Pedro Mázala afaik, when the checkpoint a big (big state), each TM writes it's data and the JM just coordinates the process
It seems to me that Flink does not perform the clean-up of the "entropised" files on s3 appropriately 😞
for instance, the job has reached checkpoint 28, but i still can see files related to
chk-7
even if the root checkpoint dir (the one with _metadata file) gets removed as expected
p
Are you using incremental checkpoints?
v
No
I've tried in the past but it seemed to me being ever-growing in size on
s3
p
Hum.. That's weird. I don't seem to find a bug related to it. have you faced any logs about issues with cleaning them?
v
any idea what log patterns should i look for?
p
Not really
v
did not look thoroughly but nothing special at first glance
p
But the rate limit issue got sorted?
v
not totally... however I've tried with entropy "in the middle of the s3 key"
going to try it with the entropy as first "folder" after the bucket
not clearly stated in aws s3 documentation but recommended by flink
Hi @Pedro Mázala The switch from
s3p
to
s3a
seems to have solved the problem (with or without the entropy injection)
so, thanks a lot for your suggestions
p
Oh! That's great news. Happy to have helped :)
v
I had a feeling that i'd read somewhere that s3p was the recommended approache. But now, after your suggestion and some extra "googling", it's everywhere that s3p is mostly for dev/testing purposes and s3a should be used indeed
🙌 1
m