Hey all, I am developing flink job which backfill ...
# troubleshooting
k
Hey all, I am developing flink job which backfill historical data and keep some values of data in state. Now backfill data size is almost 2 TB of state size. Currently I am using rocksDB state backend and it is giving my so much resources expense. Can anyone suggest cost effective approach for this much of huge data?
p
this is quite some state indeed! The usecase rings some questions that may drive the direction. 1. Which Flink version are you using? 2. How are you handling this state? I mean, what do you store there? do you need everything stored indeed? Can you play with some TTL? 3. Do you need the state to do which kind of computation? Would a external service be able to handle this state? 4. Have you heard of disaggregated state?
k
1. 1.19 version 2. Can't play with TTL also state is highly optimized like I have stored minimal value in state 3. I need state to store historical data and compute values on realtime 4. No let me look at it
m
@Keyur Makwana did you find the solution?, also how long did it took to backfill 2tb of data. Did you run the job in batch mode?
k
Solved using realtime mode only, by developing in house DB compatible with Flink that took me only 2 hour to backfill complete data and then switch to realtime model
🙌 1