Hi, is there a way to create a keyedstream without...
# troubleshooting
s
Hi, is there a way to create a keyedstream without shuffling the data using keyby(), e.g. I just want to read the data in each kafka partition, and does partial aggregation individually for each kafka partition. Thanks
o
There is the method
reinterpretAsKeyedStream
, but use with caution: Flink key groups are not the same as Kafka partitions. They can be similar enough to make the method useful though, but that depends on your use case. This method should not be used when there is a chance that a key is distributed into multiple partitions, as happens when you add partitions to a topic. For more information: https://stackoverflow.com/a/72629308/3169109 https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/stateful-stream-processing/#~text=Key%20Groups%20are%20the%20atomic,one%20or%20more%20Key%20Groups.
j
I think you need to do the partial aggregation without calling keyBy on the stream. You can use a ProcessFunction with a normal java HashMap to store the aggregators for each key you encounter and use CheckpointedFunction Interface to write the contents to operator state on checkpoints.
s
looks likie reinterpretAsKeyedStream is not an option, as the key distribution need to conform with what Flink internal do. so if we directly consumed from Kafka, the key distribution will not be right.
I am also thinking the operator state approach.
One thing i am not sure for the operator state is the recovery part: in case the job get restarted, or we scale up number of kafka partitions, how do we guarantee the current operator instance that process a certain kafka partition (say partition 1) before will still get the state of the previous operator instance that process the same partition before the restart/scaling?
j
You cannot ensure that. But if you do partial aggregation and then key by to merge the partial results, it should not be a problem.