@RunLLM How to configurate load balancing for namespaces between brokers? Topic per namespace. Now I have one broker without any load

To configure load balancing for namespaces between brokers in Pulsar, ensure your namespace is sharded into multiple bundles, as topics are assigned to brokers at the bundle level, not the topic level. When you create a namespace, specify the number of bundles (e.g.,

--bundles 16

) to allow topics to be distributed across multiple brokers. If you have only one bundle, all topics in that namespace will be handled by a single broker, causing imbalance. If you already have a namespace, you can split existing bundles using the

pulsar-admin namespaces split-bundle

command to increase the number of bundles, which enables Pulsar to redistribute load to idle brokers. Automatic bundle splitting and load shedding are enabled by default, but you can tune thresholds in

broker.conf

for more aggressive balancing. For details, see Load balance across brokers and Bundle splitting. Would you like more detail on bundle splitting or broker configuration? Sources: - Load balance across brokers - Bundle splitting 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how to manually trigger load balancing?

To manually trigger load balancing in Pulsar, you can unload specific bundles from a broker using the

pulsar-admin namespaces unload

command. This releases the bundle from its current broker, allowing Pulsar to reassign it to another broker, which helps redistribute load. For example:

bin/pulsar-admin namespaces unload <tenant>/<namespace> -b <bundle-range> -d http://<broker-url>:8080

For step-by-step instructions and verification, see Unload bundles. Would you like more detail on identifying which bundles to unload or how to check the results? Sources: - Unload bundles 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM what about this warning? Draining hash 26602 not found in the tracker for consumer Consumer ?

This warning means the broker tried to update or clear a "draining hash" (a hash range being reassigned between consumers in a Key_Shared subscription), but the hash was not found in the internal tracker for that consumer. This can happen if the hash was already cleared or if there was a race condition during consumer or hash state changes. It is generally informational and not harmful, as the system is designed to handle such cases gracefully. For more context, see the code and comments in DrainingHashesTracker.java. Would you like more detail on when this warning appears or how draining hashes work? Sources: - DrainingHashesTracker.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM as i see, with auto load balancig I don't see any balance. One broker always stay free without any bundles

If one broker always stays free without any bundles, it's likely due to insufficient bundle splitting or misconfiguration. Pulsar assigns topics to brokers at the bundle level, so if your namespace has only one or a few bundles, all traffic may remain on a single broker, leaving others idle. Increasing the number of bundles (either at namespace creation or by splitting existing bundles) allows Pulsar to distribute load across all brokers automatically. Check your namespace's bundle count and use

pulsar-admin namespaces split-bundle

to increase it if needed. Also, ensure automatic bundle splitting and unloading are enabled in your

broker.conf

. For more, see Load balance across brokers and Bundle splitting. Would you like more detail on how to check and split bundles? Sources: - Load balance across brokers - Bundle splitting 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM every namespace splited to 128 bundles

If every namespace is already split into 128 bundles and you still see one broker without any bundles, the issue is likely with load balancer configuration or thresholds, not bundle count. Pulsar should distribute bundles across all brokers if enough bundles exist, but misconfigured load balancer settings (e.g., thresholds, strategies, or disabled features) can prevent this. Check that

loadBalancerSheddingEnabled

is true, and review your

loadBalancerLoadSheddingStrategy

and related thresholds in

broker.conf

. Also, ensure new brokers are not excluded from assignment due to configuration or recent changes. For troubleshooting, see Load balance across brokers and Bundle assignment. Would you like more detail on which configuration parameters to check? Sources: - Load balance across brokers - Bundle assignment 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder lowerBoundarySheddingEnabled=false loadBalancerLoadPlacementStrategy=org.apache.pulsar.broker.loadbalance.impl.LeastLongTermMessageRate loadBalancerBrokerThresholdShedderPercentage=10 loadBalancerAverageResourceUsageDifferenceThresholdPercentage=10 loadBalancerMsgRateDifferenceShedderThreshold=50 loadBalancerMsgThroughputMultiplierDifferenceShedderThreshold=4 loadBalancerHistoryResourcePercentage=0.9 loadBalancerBandwidthInResourceWeight=1.0 loadBalancerBandwidthOutResourceWeight=1.0 loadBalancerCPUResourceWeight=1.0 loadBalancerDirectMemoryResourceWeight=0 loadBalancerBundleUnloadMinThroughputThreshold=10 namespaceBundleUnloadingTimeoutMs=60000 loadBalancerAvgShedderLowThreshold = 15 loadBalancerAvgShedderHighThreshold = 40 loadBalancerAvgShedderHitCountLowThreshold = 8 loadBalancerAvgShedderHitCountHighThreshold = 2 maxUnloadPercentage = 0.2

Your configuration uses the

ThresholdShedder

strategy for load shedding and

LeastLongTermMessageRate

for bundle placement, with thresholds set to trigger unloading when a broker's usage exceeds the cluster average by 10%. This setup should work for most cases, but if a broker remains idle, it may be due to the placement strategy not considering resource usage weights, or the thresholds being too high for your cluster's load variance. Consider switching

loadBalancerLoadPlacementStrategy

to

LeastResourceUsageWithWeight

for better resource-based distribution, and ensure

loadBalancerSheddingEnabled

is true. For more on these parameters and their effects, see ThresholdShedder and Bundle placement strategies. Would you like a step-by-step guide to tuning these settings? Sources: - ThresholdShedder - Bundle placement strategies 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.