Amit Peshwani
09/29/2024, 7:06 AMdata/{unique-id}/record.json . Estimated record size is 10-150kb. We are using synchronous S3 PUT requests for updating/creating the records and have observed that it creates backpressure to the entire flink application and slows the event-reading from the source. Eventually this increases the latency as well for updating record.json when there is a heavy load. We are also under the allowed limit of 3500 requests per second.
Can anyone share any insight if this is the wrong way of sinking the records to S3 or any solution that speeds up sinking process and no backpressure is created?Pedro Mázala
09/30/2024, 7:53 AM