Hey folks, We are working with flink streaming pip...
# troubleshooting
a
Hey folks, We are working with flink streaming pipeline 1.19 version and created a custom S3 sink that creates/updates new record.json for each unique-id. Example:
data/{unique-id}/record.json
. Estimated record size is 10-150kb. We are using synchronous S3 PUT requests for updating/creating the records and have observed that it creates backpressure to the entire flink application and slows the event-reading from the source. Eventually this increases the latency as well for updating record.json when there is a heavy load. We are also under the allowed limit of 3500 requests per second. Can anyone share any insight if this is the wrong way of sinking the records to S3 or any solution that speeds up sinking process and no backpressure is created?
p
Reading from the top, what i see could improve it: • batch sink to s3 • Async write to s3 But both would require some management from your side