Vadim Vararu
05/15/2025, 8:39 AMs3p://) because of request rate limits throttling.
I've checked and there are ~3800 objects in s3, which seems to reach the hard limit of max 3500 puts/sec
Is there any way to circumvent this limitation? Look like a blocker to use Flink for big parallelism stateful use cases 😞
To be mentioned that I use RocksDb as state backendVadim Vararu
05/15/2025, 8:41 AMPedro Mázala
05/15/2025, 9:15 AMs3a protocol and enabling entropy injection to solve this issue.Vadim Vararu
05/15/2025, 9:16 AMVadim Vararu
05/15/2025, 9:18 AMPedro Mázala
05/15/2025, 9:18 AMVadim Vararu
05/15/2025, 9:18 AMVadim Vararu
05/15/2025, 9:18 AMPedro Mázala
05/15/2025, 9:19 AMVadim Vararu
05/15/2025, 11:10 AMVadim Vararu
05/15/2025, 1:10 PMVadim Vararu
05/15/2025, 1:11 PMchk-7Vadim Vararu
05/15/2025, 1:12 PMPedro Mázala
05/15/2025, 1:14 PMVadim Vararu
05/15/2025, 1:14 PMVadim Vararu
05/15/2025, 1:15 PMs3Pedro Mázala
05/15/2025, 1:17 PMVadim Vararu
05/15/2025, 1:18 PMPedro Mázala
05/15/2025, 1:18 PMVadim Vararu
05/15/2025, 1:18 PMPedro Mázala
05/15/2025, 1:19 PMVadim Vararu
05/15/2025, 1:19 PMVadim Vararu
05/15/2025, 1:20 PMVadim Vararu
05/15/2025, 1:21 PMVadim Vararu
05/16/2025, 11:19 AMs3p to s3a seems to have solved the problem (with or without the entropy injection)Vadim Vararu
05/16/2025, 11:19 AMPedro Mázala
05/16/2025, 11:20 AMVadim Vararu
05/16/2025, 11:22 AMMaxime Menard
05/16/2025, 9:25 PM