@RunLLM bookie consume much more disk than show metadata. For example, in metadata i see 6gb for all topics, but ledger catalog on some bokies has 100GB. In sum for all bookies ledger storage consume 400GB (5 bookies with rf=2). GC enabled, but cleanup only few GB. Where I can see the problem?

This discrepancy is common in BookKeeper/Pulsar clusters. Bookie disk usage can be much higher than what is reported in Pulsar metadata because BookKeeper stores data in entry log files that interleave entries from many ledgers. Even if most data in a log is deleted, the file is not removed until all ledgers in it are deleted or compacted. Compaction (GC) only reclaims space when the proportion of deleted data in a log file exceeds certain thresholds, and some files may linger if they contain even a small amount of live data. This is expected behavior and is described in the BookKeeper documentation on data compaction. If GC is only reclaiming a few GB, it's likely that most entry log files still contain some active ledgers, preventing their deletion. You can investigate further by checking which ledgers are still active in each entry log file and reviewing your compaction thresholds and intervals. For more details, see BookKeeper data compaction. Would you like more detail on how to analyze entry log files or tune compaction? Sources: - BookKeeper data compaction 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM in this cluster i had many old sst files. and main part of disk they consumed

Large numbers of old SST files in the BookKeeper

locations

directory can accumulate, especially after upgrades or under high throughput, and are not always cleared by retention or TTL settings. This is a known issue and is often related to RocksDB compaction not being aggressive enough or not triggered as expected. Tuning RocksDB compaction parameters for the location index may help, but the exact settings depend on your workload and BookKeeper version. See discussion and suggestions in this issue. Would you like more detail on how to tune RocksDB compaction or identify which SST files can be safely deleted? Sources: - BookKeeper issue: Large number of SST files in locations not being cleared after upgrade 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM yes, please

To address old SST files in the

locations

directory, you should tune RocksDB compaction for the entry location index. BookKeeper now supports using a dedicated

entry_location_rocksdb.conf

file for this purpose. You can increase compaction aggressiveness by adjusting parameters like

level_compaction_dynamic_level_bytes

,

write_buffer_size

,

max_background_jobs

, and triggering manual compaction if needed. Also, ensure that

deleteRange

operations are followed by a

compactRange

to clean up SSTs, as background compaction may skip files with only tombstones. See BookKeeper PR #4555 and BookKeeper RocksDB config docs. To manually trigger compaction, you can use the BookKeeper shell or REST API, or adjust the compaction interval to run more frequently. If you are on a recent version, check that the

entryLocationCompactionInterval

is set appropriately and not disabled. For more, see BookKeeper PR #4555 and RocksDB config settings. Sources: - BookKeeper PR #4555 - BookKeeper RocksDB config docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

related to the previous answer by RunLLM, there's this change in Pulsar Helm chart to configure entry location compaction interval: https://github.com/apache/pulsar-helm-chart/pull/633

Thanks @Lari Hotari. I saw your commit and already added this setting to the config. But I also see, that than settings doesn't change anything, and only manual trigger now helped me. I set 1 hour, and nothing runned. Does pulsar has issue about it? Should I add cronjob for conpaction while it will be fixed? Thanks

there's no bug in Pulsar itself, https://github.com/apache/bookkeeper/pull/4555 is the BookKeeper issue. Most likely the problem is fixed when BookKeeper 4.18.0 comes out with RocksDB 9.9.3

Thanks for the answer. I will configurate external trigger, until we wait new release. When new bookie was released, will be new bookie to LTS version?