<@U078Y59KC3B> are there any existing issues repor...
# ask-ai
m
@RunLLM are there any existing issues reported by users where multiple brokers shed load at the same time and could it be caused by load balancer?
r
Yes, there are user-reported issues where multiple brokers shed load simultaneously, often leading to instability or prolonged unbalanced states. This behavior is typically linked to the load balancer's configuration or algorithm. For example, with the ThresholdShedder strategy, users have observed frequent and repeated bundle unloading, sometimes causing brokers to quickly become overloaded again after shedding, especially when the load is not evenly distributed or when the candidate selection logic is too restrictive or random. This can result in a cycle where brokers continuously shed and reacquire bundles, preventing the cluster from reaching a balanced state. See this issue and this pull request for detailed user reports and analysis. Would you like more detail on the root causes or mitigation strategies? Sources: - https://github.com/apache/pulsar/issues/21174 - https://github.com/apache/pulsar/pull/16059 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
we recently saw an issue where we saw brokers of an isolation group unloading bundles all at once leading to high latency and client timeouts. Post that the produce on some of the brokers on that IG was failing. This was mitigated by manually restarting brokers, is there any such issue reported in the past? @RunLLM
r
Yes, similar issues have been reported where brokers in an isolation group unload bundles simultaneously, causing high latency, client timeouts, and even produce failures. This behavior is often related to the load balancer's handling of isolation policies and bundle unloading logic. Specifically, if bundles with isolation or anti-affinity policies are not properly excluded from auto-unloading, or if the set of eligible destination brokers is too limited, it can lead to a situation where all brokers in the group try to shed load at once, resulting in instability and failures. This scenario and its mitigation (manual broker restarts) have been discussed in PIP-192 and related PRs and this issue. Would you like more detail on configuration options or recent fixes for this behavior? Sources: - https://github.com/apache/pulsar/pull/19592 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
@RunLLM what is the use of the load-report command, the bundles that it shows, which broker do they belong to?
r
The
load-report
command is used to dump the broker load-report, which provides information about the current load and resource usage of a Pulsar broker, including details about bundles managed by that broker. The bundles shown in the load report are those currently assigned to the broker where the command is executed or queried. Each bundle in the report represents a group of topics handled by that specific broker at the time of the report. For usage, see pulsarctl broker-stats load-report and Pulsar admin broker-stats load-report. Would you like more detail on interpreting the load report output? Sources: - https://docs.streamnative.io/pulsarctl/latest#-em-load-report-em- - https://pulsar.apache.org/docs/3.2.x/reference-pulsar-admin 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
m
how to see load report of a particular broker, what happens if command is executed in toolset pod @RunLLM
@RunLLM how to see load report of a particular broker, what happens if command is executed in toolset pod
r
To see the load report of a particular broker, use the
pulsar-admin broker-stats load-report
or
pulsarctl broker-stats load-report
command and specify the broker's admin service URL with the
--admin-service-url
flag. This command fetches the load report for the broker at the given URL. If you run the command in a toolset pod without specifying a broker, it will fetch the load report from the default broker configured in the toolset pod's environment, which may not be the broker you intend to inspect. See pulsarctl load-report usage and Pulsar admin broker-stats load-report. Would you like a sample command or more details on specifying the broker? Sources: - https://docs.streamnative.io/pulsarctl/latest#-em-load-report-em- - https://pulsar.apache.org/docs/3.2.x/reference-pulsar-admin 👍 0 button 👎 1 button 💡 Hint: Mention @RunLLM in the thread for followups.