https://pulsar.apache.org/ logo
Join SlackCommunities
Powered by
This message was deleted.
# general
s
This message was deleted.
a
@Hang Chen
h
a) Backlog for the topics that we produced messages to is showing 0 , however the `storageSize`(from TopicStats) is not. Shows a sizable number ~ 130 MB on those topics
Even though the backlog is 0, but the cursor’s mark-delete position still point to the last ledger of the topic, and it won’t be deleted. So the 130MB storage size maybe the last ledger’s size. You can use 
bin/pulsar-admin topics stats-internal <topic>
to check it
b) The disk space is not getting reclaimed even after almost 2 days. We have tweaked the 
gcWaitTime
& 
majorCompactionInterval/threshold
configuration properties in hopes of it clearing the disk space.
After the disk runs into read-only (95% of usage), the garbage collection on Bookie won’t work. You can make it work by following this PR: https://github.com/apache/bookkeeper/pull/3205
Even after running consumers to drain of any (hidden) backlog, the 
disk utilization %
hasn’t come down.
The root cause is the same with b)
d) We have also tried running 
clear-backlog
command from CLI at the namespace level without any help in clearing out the disks
The root cause is the same with b)
s
@Hang Chen Thank you very much for your inputs and suggestions. I see the above PR changes are in 
Bookkeeper v4.15.0
.. Would upgrade to 
Pulsar v2.11.0
address this issue since it uses 
4.15.4
? Thanks
h
Yes
thankyou 1
s
Also, 
forceAllowCompaction
seems to be a broker_server.conf property in bookkeeper project. Is that exposed as part of Pulsar configuration in 
bookkeeper.conf
?
h
Yes, you can set 
forceAllowCompaction=true
in conf/bookkeeper.conf. The difference between 
bk_server.conf
in the BookKeeper repo and 
conf/bookkeeper.conf
in the Pulsar repo is Pulsar overrides some configurations for BookKeeper.
👍 1
s
So.. After a) Upgrading to Pulsar 
v2.11.1
(Bookkeeper v`4.15.4`) b) Setting 
forceAllowCompaction
to true in conf/bookkeeper.conf I keep getting 
HTTP 500
Internal Server Error (details below)
Copy code
curl -v -XPUT <http://localhost:8000/api/v1/bookie/gc>
*   Trying 127.0.0.1:8000...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8000 (#0)
PUT /api/v1/bookie/gc HTTP/1.1
Host: localhost:8000
User-Agent: curl/7.68.0
Accept: /

* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< content-length: 21
The majorCompactionCounter keeps increasing in gc_details response
Copy code
curl -XGET <http://localhost:8000/api/v1/bookie/gc_details>
[ {
  "forceCompacting" : false,
  "majorCompacting" : false,
  "minorCompacting" : false,
  "lastMajorCompactionTime" : 1685458132776,
  "lastMinorCompactionTime" : 1685458132776,
  "majorCompactionCounter" : 1745,
  "minorCompactionCounter" : 0
}]
Any pointers on why ? I couldn't find anything in the logs
h
Copy code
500 Internal Server Error
Could you share the bookie server logs, it is a server error
s
bookkeeper-server.log file is empty and only available in one of 3 bookie pods 😭 . I 
exec
'd into a bookie pod to check for it .. Am I supposed to be looking for it elsewhere ?
h
kubectl logs -f <pod>
You can check each bookie pod
Open in Slack
PreviousNext
Powered by