Hi,
I'm running a Pulsar 4.0.2 Cluster (3 broker, ...
# generalc
ck_xnet
10/01/2025, 1:01 PMHi,
I'm running a Pulsar 4.0.2 Cluster (3 broker, 3 bookies, 3 zookeeper) with partitioned topics and I have two problems:
1. The filesystem usage on the bookies doesn't seem to go down (bk1: 100GB, bk2: 400GB, bk3: 100G). I already set a retention policy on my main namespace (2weeks, 10GB) and the metriks in Grafana report the correct topic sizes (100GB storage size and ~30GB backlog size). I get the usage on bk1 and bk3, but not the 400GB on bk2...
2. The bookie service on bk2 stops frequently with an error "io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 4194304 byte(s) of direct memory (used: 2147483648, max: 2147483648)". I can't find the setting where I can control the limit for this memory setting. I already increased the jvm allocations in the pulsar_env.sh those don't seem to correlate..
Thanks!
a
Alvaro Rodriguez
10/01/2025, 1:37 PMCan you try to increase memory size of the bookies to see if the problem is that you are hitting an OOM solved just increasing the memory to 4GiB so the GC can be properly run?
c
ck_xnet
10/01/2025, 1:52 PMThe bk2 has ~5GB memory available, I increased the memory of bk1 and bk3. Is there a Log entry if there is not enough memory for GC?
a
Alvaro Rodriguez
10/01/2025, 3:07 PMDid the OOM resolved?
Alvaro Rodriguez
10/01/2025, 3:10 PMCan you check in the bookies what files are consuming space?
Alvaro Rodriguez
10/01/2025, 3:11 PMCan you check this metric
bookie_DELETED_LEDGER_COUNTc
ck_xnet
10/02/2025, 6:49 AMthe bk02 bookie crashed again with the same error
ck_xnet
10/02/2025, 6:52 AMI checked that metric. I guess you want to know if anything gets deleted at all?
When i restarted the bookie yesterday morning, it instantly deleted 100 ledgers and the number still increased a bit until it crashed. When I restarted it in the afternoon, 7 ledgers were deleted and it went up to 24 until it crashed.
ck_xnet
10/02/2025, 7:15 AMalso the metric is called bookie_DELETED_LEDGER_TOTAL on my end 😉
👀 1
a
Alvaro Rodriguez
10/03/2025, 3:54 AMHave you connect to the Bk and can you list the files inside?
c
ck_xnet
10/03/2025, 6:46 AMYes, it's the .log files in the ledgers directory. There are around 12k files with file sizes ranging from 4k to 1.2g.
In comparison: on the other 2 bookies are ~200 and 85 files..
a
Alvaro Rodriguez
10/03/2025, 9:28 AMDo you have the logs from the GC on that bookie? Can you search for this string GarbageCollectorThread and WARN and ERRORS, maybe you can create a ticket on GH with that information to debug why the .log files are not removed
c
ck_xnet
10/03/2025, 9:42 AMyes, there are recurring INFO entries:
"Forced garbage collection triggered by thread: LedgerDirsMonitorThread",
"Garbage collector thread forced to perform GC before expiry of wait time",
"Extracting entry log meta from entryLogId: 185",
"GarbageCollectorThread-6-1 Set forceGarbageCollection to false after force GC to make it forceGC-able again"
ck_xnet
10/03/2025, 9:44 AMShould I open it as discussion, or issue?
a
Alvaro Rodriguez
10/03/2025, 10:36 AMI would open an issue, providing as much information as possible. Mention the OOM that keeps happening, and if possible, add the latest logs and those logs for the Garbage collector.
c
ck_xnet
10/03/2025, 10:46 AMalright, thank you!
4 Views