Hey all,
I am developing flink job which backfill historical data and keep some values of data in state. Now backfill data size is almost 2 TB of state size. Currently I am using rocksDB state backend and it is giving my so much resources expense. Can anyone suggest cost effective approach for this much of huge data?
p
Pedro Mázala
05/14/2025, 1:09 PM
this is quite some state indeed!
The usecase rings some questions that may drive the direction.
1. Which Flink version are you using?
2. How are you handling this state? I mean, what do you store there? do you need everything stored indeed? Can you play with some TTL?
3. Do you need the state to do which kind of computation? Would a external service be able to handle this state?
4. Have you heard of disaggregated state?
k
Keyur Makwana
05/14/2025, 1:12 PM
1. 1.19 version
2. Can't play with TTL also state is highly optimized like I have stored minimal value in state
3. I need state to store historical data and compute values on realtime
4. No let me look at it
m
Mahima Lalwani
08/04/2025, 12:03 PM
@Keyur Makwana
did you find the solution?, also how long did it took to backfill 2tb of data.
Did you run the job in batch mode?
k
Keyur Makwana
08/04/2025, 12:09 PM
Solved using realtime mode only, by developing in house DB compatible with Flink that took me only 2 hour to backfill complete data and then switch to realtime model