hello, do tuningConfig.appendableIndexSpec offheap...
# general
r
hello, do tuningConfig.appendableIndexSpec offheap get implemented? I saw https://github.com/apache/druid/issues/9967 is open but at the same time https://github.com/apache/druid/issues/10321 is closed I'm having issue with [auto]compaction of one spefic day where after a few hours, I always get out of heap or kiled by OS I alreayd tried setting very low values for "maxRowsInMemory": 50000, "maxBytesInMemory": 60000000, "maxNumConcurrentSubTasks": 2, "totalNumMergeTasks": 2, maxColumnsToMerge 40 (I have 99) all partial_dimension_cardinality jobs goes success, but when it goes into partial_index_generate whening finalizing merging phases goes out of heap space, In the start of the logs I saw some lines
Copy code
2024-09-14T11:31:13,649 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[104,857,600]
2024-09-14T11:31:13,845 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[104,857,600]
2024-09-14T11:31:13,948 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[104,857,600]
so this is indeed low, about just 300 MB but I have -Xms750m -Xmx2500m -XX:MaxDirectMemorySize=4g on the runner I have many small datasources and just 2 big ones, that's why I set the MM memory "low", otherwise I would need to scale all MM to fit the bigger