Xiaobing Xia
07/02/2024, 5:06 AM// Directly build a slot sharing group with specific resource
SlotSharingGroup ssgWithResource =
SlotSharingGroup.newBuilder("ssg")
.setCpuCores(1.0) // required
.setTaskHeapMemoryMB(100) // required
.setTaskOffHeapMemoryMB(50)
.setManagedMemory(MemorySize.ofMebiBytes(200))
.setExternalResource("gpu", 1.0)
.build();
I didn't see a way to configure the network memory. Is there anyway to do that? thanksD. Draco O'Brien
07/02/2024, 7:49 AMD. Draco O'Brien
07/02/2024, 7:50 AMD. Draco O'Brien
07/02/2024, 7:52 AMD. Draco O'Brien
07/02/2024, 7:52 AMD. Draco O'Brien
07/02/2024, 7:55 AMXiaobing Xia
07/02/2024, 5:50 PM2024-07-01 22:57:24,573 ERROR flink.realtime.processor.datasources.mysql.MySqlCdcStreamRouter [] - Error in MySqlCdcStreamRouter.processElement(): message: Buffer pool has already been destroyed., cause: null
After investigation, some one says it is because my network memory is too small, that's why I'm looking to increase the network memory size of each task slot under fine-grained resource management mechanism.
Are we saying, given this scenario, the Flink job will use more network memory than 18mb if needed?Saurabh Singh
07/03/2024, 7:43 AMXiaobing Xia
07/03/2024, 3:39 PMXiaobing Xia
07/04/2024, 2:35 PMtaskmanager.network.memory.buffers-per-channel: '8'
taskmanager.network.memory.max-buffers-per-channel: '40'
taskmanager.network.memory.floating-buffers-per-gate: '32'
it would increase network buffer size of each slot