I'd like to ask about the internals in Flink SQL. As far as I understand, the changelog is a set of changes that are stored independently, meaning these are retained at the change level. Is it possible to reduce the state by removing some changes older than X days/hours without affecting the latest changes that are relevant for the processing? If so, is there a FLIP with a potential proposal that I can watch? Or could it be achieved by using the State Processor API? I am not looking for an example but more for the direction in which this can be done. As far as I understood, TTL works at the key level and not the change level.
Namely, we receive updates multiple times a day and would like to retain only a subset of the changes. As an example, every hour we receive a new update (U+), but we plan on retaining only the latest besides the insertion and delete kinds.
Status quo: I U+ U+ U+ U+ within 24 hours
After (FLIP/State Processor API) run: becomes I U+ within 24 hours.
The main concerns are the ability to 1. manage an ever-growing state and 2. slower deserialization if all changes need to be deserialized and checked before emitting the new change.
I can also ask in the user mailing list, what's the best approach? Thank y'all.