@RunLLM When using pulsar, Netty direct memory OOM occurred in bookkeeper, which triggered the exit logic of bookie. However, it was stuck in rocksdb when exiting, and the time was as long as 48 minutes. What could be the problem?
r
RunLLM
07/17/2024, 10:11 AM
The issue you're experiencing with Netty direct memory OOM in BookKeeper, followed by a prolonged exit time due to RocksDB, could be related to several factors:
1. Backpressure Configuration: According to Bookie OutOfDirectMemoryError, the problem might be related to backpressure configuration. Under heavy load, Bookie might not process data fast enough, causing data to accumulate in memory. Configuring backpressure on both client and server sides can help mitigate this issue.
2. RocksDB Batch Size: As per Allow to set max operation numbers in a single rocksdb batch, memory usage in RocksDB is related to batch size. Large batches can consume significant memory, leading to OOM errors. Limiting the batch size can help control memory usage.
3. RocksDB IOException Handling: The issue might also be related to how RocksDB handles IOExceptions. According to rocksdb IOException cause checkpoint flush process keep running and failing continuously, continuous unsuccessful flush attempts due to IOExceptions can lead to prolonged exit times. Properly handling these exceptions and possibly triggering a shutdown can help.
Would you like more details on any of these points?
Sources:
- Bookie OutOfDirectMemoryError
- Allow to set max operation numbers in a single rocksdb batch
- rocksdb IOException cause checkpoint flush process keep running and failing continuously 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.