This message was deleted.
# generals
Slackbot
11/24/2023, 6:29 PMThis message was deleted.
y
Yu Wei Sung
11/24/2023, 6:54 PMyou need to look at vmstat to find where the cpu busy on
j
Jerome B
11/24/2023, 7:01 PMI suppervise every process, it’s bookkeeper (only one node) which make cpu busy. On the other node cpu is low.
l
Lari Hotari
11/24/2023, 8:45 PMWell if metrics don't show the cause, it's worth doing some sort of profiling for the process. jstat gives a thread dump and a few subsequent thread dumps could reveal something. JFR profiler is built-in in the JVM and is very powerful. AsyncProfiler is also an option. I have some tooling at https://github.com/lhotari/k8s-diagnostics-toolbox which can be used on a k8s node to profile any Java process, also in containers without shell access.
Lari Hotari
11/24/2023, 8:47 PMHere are some other scripts:
https://github.com/lhotari/pulsar-contributor-toolbox/tree/master/scripts
Lari Hotari
11/24/2023, 8:48 PMUseful in k8s environment for docker containers with JDK tools and a shell such as in the Pulsar docker image
Lari Hotari
11/24/2023, 8:49 PMFor example, this can do JFR profiling: https://github.com/lhotari/pulsar-contributor-toolbox/blob/master/scripts/jfr_profile_pod.sh
j
Jerome B
11/25/2023, 9:54 AMI didn't know about JFR, thanks for sharing, I'll try that.
l
Lari Hotari
11/25/2023, 10:10 AMYou can also use AsyncProfiler's tooling to create a html/svg FlameGraph out of the .jfr dump. JMC has FlameGraph view, but it's not so usable as AsyncProfiler's html view.
The tool is in converter.jar that can be downloaded from https://github.com/async-profiler/async-profiler/releases/tag/v2.9
The converter is also available via ap-loader.jar
https://github.com/jvm-profiling-tools/ap-loader#converter
3 Views