Hello team, for fine grained resource management,...
# troubleshooting
x
Hello team, for fine grained resource management, from the doc:
Copy code
// Directly build a slot sharing group with specific resource
SlotSharingGroup ssgWithResource =
    SlotSharingGroup.newBuilder("ssg")
        .setCpuCores(1.0) // required
        .setTaskHeapMemoryMB(100) // required
        .setTaskOffHeapMemoryMB(50)
        .setManagedMemory(MemorySize.ofMebiBytes(200))
        .setExternalResource("gpu", 1.0)
        .build();
I didn't see a way to configure the network memory. Is there anyway to do that? thanks
d
I dont think direct configuration of network memory is supported within a SlotSharingGroup via the builder method.
memory is managed implicitly via the JVM
You can indirectly affect this via task manager memory config and network buffer configuration
ie taskmanager.memory.process.size
taskmanager.netowrk.memory.min/max and taskmanager.network.memory.fraction but these require careful calibration for your overall workloads
x
thanks for the explanation. Here's my network memory allocation at my job: you can see each task slot only get allocated around 18mb network memory. And recently I had following exception from my ProcessFunction when emitting records to side output:
Copy code
2024-07-01 22:57:24,573 ERROR flink.realtime.processor.datasources.mysql.MySqlCdcStreamRouter [] - Error in MySqlCdcStreamRouter.processElement(): message: Buffer pool has already been destroyed., cause: null
After investigation, some one says it is because my network memory is too small, that's why I'm looking to increase the network memory size of each task slot under fine-grained resource management mechanism. Are we saying, given this scenario, the Flink job will use more network memory than 18mb if needed?
s
@Xiaobing Xia What is the total memory of the pod/system you are using to run this? Are the things adding up? Network memory might be getting exhausted to establish channels with each available task slot. Otherwise, you can give it a try to reduce the heap memory to come out of this problem?
x
My total memory allocation is in above screenshot? unless I misunderstood the question. It is 61637mb total heap memory , separated into 10 task slots, each slot is 6144mb. Unfortunately I can't reduce the heap memory otherwise my system will OOM. If my network memory is exhausted, not sure how I can increase the network memory allocation for each fined grained resource group.
the answer is tuning following configs:
Copy code
taskmanager.network.memory.buffers-per-channel: '8'
taskmanager.network.memory.max-buffers-per-channel: '40'
taskmanager.network.memory.floating-buffers-per-gate: '32'
it would increase network buffer size of each slot
✅ 1