I’m using the dynamodb sink which is subclassed from AsyncSinkBase - is it okay if I set

maxTimeInBufferMS

to 500?

You should be able to tune the

maxTimeInBufferMS

to 500ms without issue. The tradeoff here is that calls to DDB will be more frequent, and in smaller batches, which theoretically will have an impact on the maximum throughput of the job.

How can I measure the time taken for events to pass through Flink, and (hopefully) find out what I need to tweak to get these realtime events written to dyanmodb

One way is to write the Event time as a field in the DDB entry. I am not aware of a way to measure it natively in Flink 👀 Curious, what is your expected E2E latency from publishing to the Kinesis stream to the record appearing in DDB?

Hi Hong, once I set the

maxTimeInBufferMS

to 500 things started running smoothly. I think the default was 5000 which is why it took a little while for things to be written to dynamodb in testing where there is less throughput (e.g the batch size is 25, which will be reached faster than 500ms in prod so I guess this setting was just to get things flowing faster in dev only)

I see, good to hear the config helped. It would be worth doing some parameter tuning for production then!

and I also made the buffer setting an application property so now I can change it without redeploying 🙂