Hi Team, Need help to understand why the Flink state keeps growing for any Table Sinks or Joins we do on Tables. We are using Hudi as table source and performing a streaming read. In below screenshot, the Join has grown to 1.39GB Can we use table API to just lookup and not have it store in the state and also join based on lookup ?

what is your source and what type of join are you performing?

source is also a Hudi streaming enabled table

I believe this question is better suited for the hudi-flink slack channel as it needs information on how hudi performs the joins and if there is anything more that can be configured to help with this. But speaking strictly on the flink side, the state of the joins needs to be part of the checkpointing, so if the size of the tables keeps growing the same will happen with the state.

Thanks @Giannis Polyzos Got it. In Flink can we keep state for only last n days of data ? These are big Data Tables, and keeping the entire table in state would not work. The joins for us will work for last n days of data as well as we expect all related events to be within the n days timeframe. Appreciate any references if you can share. Thanks

If you are using Flink sql you can set https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-exec-state-ttl to expire the state after some time. But there is a tradeoff here with result completeness as you need to make sure the state is no longer needed. Again though i would suggest also asking in the hudi slack channel, because there might be some some extra configurations on the connector that im not aware of and can help. If you are using rocksdb as the state backend incremental checkpoints might also help reduce the size