Hello, I have a Flink job that has very heavy reco...
# troubleshooting
u
Hello, I have a Flink job that has very heavy records (each record is approximately 500MB, for example). What are the best practices for managing such large records in a Flink job? For example, I tried using this Flink configuration: taskmanager.runtime.large-record-handler But it caused me a lot of errors (collecting in ProcessFunction took 1 hour). What are the best practices for handling this case?
s
having 500MB record itself is a design issue.
b
If it is just for streaming ... nothing to worry about, but if you do aggregation. it will have issue on rocksdb. So better don't
u
Ok i dont really have alternative. I have algorithm that must get these matrices as input, the alternative is doing many requests for the heatmap from external storage (s3) - this is not working. So i am doing the join in the flink and collecting the joined heatmaps (list of 1000 matrices of 401x401 - this is the pojo). This is a flink batch job that runs one a day
@Blast Blast @Sucheth Shivakumar
s
does it read from kafka ?
u
I read heatmaps from s3 using file source, each heatmap is 401x401 of floats Then i have a keyedcoprocess function and then i am collecting pojo that have list of 1000 matrices of 401x401 - there is the big record problem, And again i dont have better alternative then having that large object, because the algoritm must have all of them at once, thanks!!
b
each record 500mb ?? i thought you said 1 topic 500mb
what binary encoding are you using ? you shouldn't find a way to compress it. Default is snappy.
and why you store image bytes in record. It's wrong way, you should store s3 path only
u
I can't pass just the reference to the matrix, because my algorithm needs the entire matrices (list of 1000 matrices of 401x401), I tried to use operator chaning (without keyby, suffle, splitting) and that big delay of 1 hour fixed, Thanks!