KeyedBroadcastProcessFunction or CoProcessFunction? what is the difference and when to use them while joining streams?
t
Tsering
01/12/2023, 3:01 PM
A KeyedBroadcastProcessFunction is used to process elements of a stream, where one stream is broadcasted to all the elements of another keyed stream. It is used when we have a side input (broadcast stream) that is to be used by the main input stream. The broadcast stream is used to enrich the main input stream and is typically small in size.
A CoProcessFunction is used when we have two input streams and we want to join them based on a key. It is used to process elements of both streams simultaneously, and it can also be used to implement stateful stream-stream joins.
In summary, use KeyedBroadcastProcessFunction when you have one side input stream (broadcast stream) that you want to use to enrich the elements of a keyed main input stream. Use CoProcessFunction when you have two input streams and you want to join them based on a key.
m
mgu
01/12/2023, 3:31 PM
Thanks for you reply @Tsering! Almost clear, but I am not sure but about _"process elements of both stream simultaneously_": you mean I can process elements of both stream while sharing the state of the
CoProcessFunction
between the
processElement1
and
processElement2
invocations, whereas it doesn't happen with the broadcast stream. correct?