https://pulsar.apache.org/ logo
Join Slack
Powered by
:wave: happy friday! I filed an issue yesterday, ...
# general
k
👋 happy friday! I filed an issue yesterday, seems like there might be a memory leak with pulsar key-based batching in the pulsar go client producer. Posting in #C5Z4T36F7 to raise awareness, thank you! 🙇 https://github.com/apache/pulsar-client-go/issues/1423 🤔
👀 1
m
Can you take a heap-profile?
👀 1
k
cc my team mates: @Trey Hakanson and @Grace
t
Hey there @merlimat, thanks for taking a look at this! Would a heap profile from 
pprof
be sufficient? We generally use DataDog's APM tooling for profiling, but as we've been digging more into this issue we've found that it doesn't seem to be very accurate when it comes to heap live objects and allocations
m
Would a heap profile from 
pprof
be sufficient?
Yes, it should show the refs to the the objects in mem. That would be the quickest way to see exactly where the memory is going
🫡 1
k
I'm on it, hopefully I can share something by end of day today
https://github.com/apache/pulsar-client-go/issues/1423#issuecomment-3326019406 Updated the issue. TLDR: from our pprof heap profiles, the go heap is reasonable, at 80mb in-use space whereas at the same time, 850mb in resident memory by the go process. After we stop publishing messages, this resident memory remains the same at 850mb, and when we start publishing again, the resident memory grows and doesn't fall. This makes us think the problem resides in cgo zstd message compression.
This is a heap profile that we extracted when the resident memory is bloated / leaking:
heap_3_41pm_no_cloud_nodes_pod_0.out
this is 
flamegraph?si=inuse_space
Issue has been solved and closed https://github.com/apache/pulsar-client-go/issues/1423#issuecomment-3336132565 Thanks all for the help! Some good learnings related to golang pprof, cgo, and pulsar compression here.
🎉 3
🙌 2
2 Views
Open in Slack
PreviousNext
Powered by