Hi guys, we are using all default settings for boo...
# generaly
Yang Ou
06/03/2024, 5:55 AMHi guys, we are using all default settings for bookies regarding its disk gc. Our maximum backlog size is only 500gb ， and we have 12 nodes each with 3tb storage. However we keep seeing disks getting filled up. And logs complain about disk usage so it had to pause major gc.
h
Hang Chen
06/03/2024, 9:16 AMWhich version of your bookie? Would you please share the bookie logs so that I can check what happens
Hang Chen
06/03/2024, 9:18 AMTrigger force gc may also be skipped because disk usage is almost full. There is one parameter to unlock this skip behavior
https://github.com/apache/pulsar/blob/274a3cb66b80f516b13cb1c5dfe87619fac29f98/conf/bookkeeper.conf#L180-L184
y
Yang Ou
06/03/2024, 10:42 AMHi Thanks for replying, we are using pulsar 2.10.0.7
Yang Ou
06/03/2024, 10:47 AMThis has been the pattern
Yang Ou
06/03/2024, 11:34 AMI'm asking my team to share the log. But it's quite big ~700M
Yang Ou
06/03/2024, 11:35 AMMay be I can share logs for the past two days
h
Hang Chen
06/03/2024, 3:08 PMok
Hang Chen
06/03/2024, 3:16 PMHang Chen
06/03/2024, 3:18 PMFrom the bookie garbage collection log, I found your minor compaction happens every two days. and when the minor compaction is finished, it should remove a lot of entry log files and the disk usage should decrease a lot. I wonder if you changed the compaction related parameters?
Hang Chen
06/03/2024, 3:23 PMPulsar 2.10.0.7 uses BookKeeper 4.14.4, I suggest you upgrade BookKeeper to the latest version 4.16.5, which includes many garbage collection related bug fixes and improvements
y
Yu Wei Sung
06/03/2024, 8:45 PMit seems like there are 2bookie disk usage never decrease. any retention policies? any system topic?