Hey guys! Does anyone have a good explanation for the use-case of the

KeyBy

operator? I have a Kinesis stream with 350+ event types, each event is eventually streamed into a different sink (using side-outputs) Is it a good idea to use

KeyBy

before the

process

function that tags the events? Does it have any positive effect on the pipeline in terms of durability/performance?

In terms of performance, the KeyBy operator improves the pipeline's throughput because it allows Flink to perform local data exchange in the event stream, thus maximizing the locality of the forwarded data. In addition, when using Flink's Window operator, the KeyBy operator can be optimized for the size of the window, which also helps improve the performance of the pipeline.

That’s awesome to hear, thank you! I’ll read about the window operator now 🙂