Hello everyone! Is there anyone who can give me so...
# randomj
Josip Marić
07/06/2022, 3:06 PMHello everyone! Is there anyone who can give me some input on implementing state migration(by which I mean transferring a state from one operator instance to another to achieve better load balance)?
a
Alexander Fedulov
07/06/2022, 4:50 PMI assume you are already familiar with this page?
https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/
Do you have a concrete issue/question with using it?
j
Josip Marić
07/07/2022, 10:09 AMIt's not an issue but an upgrade to Flink I'd like to implement. I know Flink still can't transfer the state from one operator instance to another during processing. However, I'm not certain how complex would it be to implement it, therefore, would like to have a discussion with someone who is an expert in the area.
a
Alexander Fedulov
07/07/2022, 10:15 AMWhat is the use case?
Alexander Fedulov
07/07/2022, 10:17 AMAnd then you are talking about operators, do you imply moving the OperatorState or also about moving keyed state?
j
Josip Marić
07/07/2022, 10:24 AMThe goal is to be able to achieve better load balance in case of a data skew, which is an actual problem when using keyed stream. Therefore, I'm mainly talking about the keyed state.
Josip Marić
07/07/2022, 10:26 AMUltimately, I'd like to transfer the complete operator instance from one task manager to another.
2 Views