Hello party people!
Using Flink SQL, I need to sink values from a kinesis stream to another kinesis stream with a UDTF in between.
The source kinesis stream has 4 shards, but I want to spread over many task slots so my UDTF can run as manny concurrent calls as possible.
Having a rough time getting a shuffle to take place between source and UDTF that doesn't convert the stream to update mode.
This means I can't use normal strategies such as
GROUP BY
, etc..., even with an intermediate view- since kinesis is an append only sink.
Interested to hear how other folks are working around this sort of issue!