This message was deleted.
# troubleshootings
Slackbot
09/23/2022, 12:37 PMThis message was deleted.
g
Gian Merlino
09/23/2022, 4:34 PMi'm not aware of anything that would increase ingest memory requirements, although there may be something i'm not aware of that happens in some case…
Gian Merlino
09/23/2022, 4:36 PMamount of direct memory usage during ingestion is pretty sensitive to tunings like number of tasks, max columns to merge, etc. it's possible the calculation of these changed in some way
Gian Merlino
09/23/2022, 4:37 PMOne common reason I've seen direct buffer memory max out is if too many columns are being merged at once; setting
maxColumnsToMergeGian Merlino
09/23/2022, 4:37 PMthe way the setting works is, let's say you have 30 columns per segment, and for each segment you have 110 intermediate persists. during segment generation, we merge all of those intermediate persists. that's 110 * 30 = 3,300 total to merge. each column has memory requirements of about 5KB on heap and 100KB direct (off heap). so that'll require about 16.5MB on heap and 500MB off heap. if you want to limit it, you can set maxColumnsToMerge = 1500 and it will use only about half that.
👍 1
Gian Merlino
09/23/2022, 4:39 PMbtw, the MSQ task runtime (new in 24) sets this automatically based on available memory. it's meant to be more self-tuning than the native batch runtime
e
eddie n
03/26/2023, 3:33 PM@Gian Merlino Thanks! This solved my Direct buffer memory OOM from compact task. Set it to 30 and then succeeded. However, it took too long (1hr 40min)
still wondering how to calculate appropriate
maxColumnsToMerge