I'd like to ask about the internals in Flink SQL. ...
# troubleshooting
a
I'd like to ask about the internals in Flink SQL. As far as I understand, the changelog is a set of changes that are stored independently, meaning these are retained at the change level. Is it possible to reduce the state by removing some changes older than X days/hours without affecting the latest changes that are relevant for the processing? If so, is there a FLIP with a potential proposal that I can watch? Or could it be achieved by using the State Processor API? I am not looking for an example but more for the direction in which this can be done. As far as I understood, TTL works at the key level and not the change level. Namely, we receive updates multiple times a day and would like to retain only a subset of the changes. As an example, every hour we receive a new update (U+), but we plan on retaining only the latest besides the insertion and delete kinds. Status quo: I U+ U+ U+ U+ within 24 hours After (FLIP/State Processor API) run: becomes I U+ within 24 hours. The main concerns are the ability to 1. manage an ever-growing state and 2. slower deserialization if all changes need to be deserialized and checked before emitting the new change. I can also ask in the user mailing list, what's the best approach? Thank y'all.
πŸ‘€ 1
j
It sounds like you want something changelog compaction. One way you could get that is by storing the changelog in Kafka and compacting the topic. It may have more messages than you like and in some cases, it may have just one +U (instead of the I and final +U). Where are you looking for the compaction to happen?
a
Yes, that's already before the job, we can freely reply from the compaction in Kafka, the idea was optimising the changelog for long-running job in case of high throughput with low-cardinality primary keys (milions of events for few thousand keys) . I am still experimenting a bit more with a keyed process function to avoid or at least manage the rate of updates and we most likely relying on windowing for mitigating such.
j
If there is any aggregation happening, you may want to look at https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/tuning/#minibatch-aggregation. I have not tried it myself yet, but that'd be one thing to consider.
In terms of implementing it yourself, that sounds like how the Upsert Kafka connector can buffer writes. https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/upsert-kafka/#sink-buffer-flush-max-rows If you look at how that is implemented, you may be able to borrow some ideas/code!
a
Thanks, I'll have a look into both. About the former, I'd need to check how to extend the mini-batch for further metrics collection and the latter we don't use Kafka directly but there is a wrapper around it (http=based), I will check the internal implementation out of curiosity. Again, thanks for the insights!
βœ… 1