https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
d
Scaling up is the easy part for bookies; the K8s operators are able to automatically allocate new PVC inside K8s, so as long as your storage provisioner has the correct permissions, it can easily allocate new EBS volumes, etc.
s
that makes sense. is automatic scaling of broker nodes a thing with pulsar?
d
Yes, a broker auto scaler is available as part of our managed solution.
s
i see in my case i'm just deploying in aws. i assume autoscaling i should be handling automatica scaling with vpa? i'm gonna be using the streamnative pulsar operator.
l
btw. There's basic HPA support for scaling brokers and proxies in the Apache Pulsar Helm chart, https://github.com/apache/pulsar-helm-chart/pull/262 . I haven't used it myself, but perhaps @Frank Kelly could share some experiences.
đź‘Ť 1
f
Yeah Proxy autoscaling is super simple for HPA as there is no state. Broker autoscaling works really well for us (we key off of CPU). The only challenge is figuring out the Shedding algorithm and the parameters. Happy to share the params that we use if it helps. For our use case (more streaming than events) we have to be careful not to shed too much load too quickly otherwise the clients will error out. Also FYI quite a few fixes and improvements for load balancing / load shedding have gone into 2.10 and above so I personally would be looking at a more recent version for those fixes.
đź‘Ť 1
But in summary HPA autoscaling of proxy and broker works quite nicely (but needs some tweaking and load testing as YMMV).
s
have either of you gents tried the datastax pulsar operator. they seem to be focused on automatic scaling. not just the brokers but bookies as well. https://www.datastax.com/blog/introducing-the-kubernetes-autoscaling-for-apache-pulsar-operator-and-the-kaap-stack.
from their blog: "Removing a bookie from the cluster is an entirely different matter. It’s known as a process of decommissioning. The basic steps are to first remove the instance from active broker interactions. Then all the data stored on the instance needs to be drained to other instances and the brokers need to be made aware of where that data goes." "When the team designed KAAP, they knew a better way to approach the scaling challenge. Instead of using an instance of each broker’s CPU load, they hooked into Pulsar’s internal load balancer. Wait—did you know Pulsar has an internal load balancer?! When KAAP sees that the Pulsar load balancer is having trouble finding brokers to assign topic bundles to, it will scale up the number of brokers to handle the load. When the operator sees that the CPU usage of all brokers is low, it will scale down the number of brokers to save resources."
f
KAAP is on my list to try - the one thing to remember is that adding a bookie is easy but decommissioning involves copy all of it's data somewhere to maintain the quorum (assuming you have a quorum > 1). It's an odd thing to think that scaling down can involve increased load. So definitely should be tested and basically you'll need to be sure that the disk size is enough to handle all the copies when you scale down to the minimum. But worth trying for sure.
d
Both of you are correct. Scaling down bookies is much harder and must be addressed carefully, given the high possibility of data loss if done incorrectly. That is why we have been slightly slower in rolling out the bookie auto-scaling features in the SN operators. We wanted to be more diligent in this area.
s
this seems to be a positive for datastax pulsar operator over steamnative. however the project looks to be much newer and maybe less active. "When KAAP sees an abundance of open storage on a Bookkeeper node, the node is automatically scaled down (decommissioned) to free up volume usage and reduce storage costs. This scale-down is done in a safe, controlled manner which ensures no data loss and guarantees the configured replication factor for all messages."
g
@Frank Kelly I would be interested in your auto shedding params, we have been trying to tune them internally for our use case .. would love to compare
f
@Girish Sharma Again we are using Pulsar 2.10.3 currently
Copy code
loadBalancerAutoBundleSplitEnabled: true
                # Default (until Pulsar 2.10) is "org.apache.pulsar.broker.loadbalance.impl.OverloadShedder"
                loadBalancerLoadSheddingStrategy: "org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder"
                # Need to allow Pulsar to split up namespace bundles to aid in load balancing. This value must be greater than defaultNumberOfNamespaceBundles above. (Default: 128)
                loadBalancerNamespaceMaximumBundles: "256"
                # Increase the frequency of load shedding (Default: 30)
                loadBalancerSheddingIntervalMinutes: "1"
                # Prevent the same topics from being shed and moved to other brokers more that once within this timeframe (Default: 30)
                loadBalancerSheddingGracePeriodMinutes: "30"
                # The broker resource usage threshold used to trigger bundle offload from broker (Default: 10)
                # Smaller value so that we are more likely to balance across brokers.
                loadBalancerBrokerThresholdShedderPercentage: "{{ PulsarBrokerLoadBalancerThresholdShedderPercentage }}"
                # Update load balancer statistic reports more frequently (Default: 10)
                loadBalancerReportUpdateThresholdPercentage: "1"
                # Smaller value so that we are more likely to base balancing decision on current load rather than historical load (Default: 0.9)
                loadBalancerHistoryResourcePercentage: "0.0"
                # We autoscale on CPU so Pulsar's measure of load should be solely based on CPU - however it does not always work so have bandwidth as backup.
                # Pulsar load balancing (shedding) is initiated if ANY of these metrics is loadBalancerBrokerThresholdShedderPercentage percent above the cluster average.
                loadBalancerCPUResourceWeight: "3.0"
                loadBalancerBandwithInResourceWeight: "1.0"
                loadBalancerBandwithOutResourceWeight: "1.0"
                loadBalancerMemoryResourceWeight: "0.0"
                loadBalancerDirectMemoryResourceWeight: "0.0"
                # Bundle unload minimum throughput threshold. Measures minimum to be unloaded. (Default: 10 MB)
                loadBalancerBundleUnloadMinThroughputThreshold: "{{ PulsarBrokerLoadBalancerBundleUnloadMinThroughput }}"
Some notes • Since we only use HPA scaling based on CPU then our primary weight for parameters is 
loadBalancerCPUResourceWeight: "3.0"
The value is above 1 because we deploy Pulsar without CPU limits (to avoid throttling) and Pulsar seems to pickup NOT the CPU requests but the number of CPUs on the node - which is large. • 
loadBalancerBrokerThresholdShedderPercentage
is parameterized - for our smaller clusters it is 
1
and for larger clusters 
5
- again with the goal of doing SOME offloading but not a lot • 
loadBalancerBundleUnloadMinThroughputThreshold
is similarly set on a per cluster basis - smaller clusters have a lower threshold, larger clusters a larger threshold • We care solely about current broker state (is it overloaded now) so we set 
loadBalancerHistoryResourcePercentage: "0.0"
Our HPA definition is
Copy code
kubectl get hpa platform-pulsar-broker -o yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
.
.
spec:
  behavior:
    scaleDown:
      policies:
      - periodSeconds: 600
        type: Pods
        value: 1
      selectPolicy: Max
      stabilizationWindowSeconds: 1800
    scaleUp:
      policies:
      - periodSeconds: 15
        type: Percent
        value: 100
      - periodSeconds: 15
        type: Pods
        value: 4
      selectPolicy: Max
      stabilizationWindowSeconds: 60
  maxReplicas: 4
  metrics:
  - resource:
      name: cpu
      target:
        averageUtilization: 55
        type: Utilization
    type: Resource
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: platform-pulsar-broker
Note in particular the 
scaleUp
and 
scaleDown
policies to help ensure we don't scale up unnecessarily (esp. on pulsar cluster creation) and we scale down slowly -it's probably too conservative so we will adjust that soon
g
Thanks @Frank Kelly! We are also on 2.10.4 so these would help. right now we dont use HPA, but the shedding configs are useful.
2 Views
Open in Slack
PreviousNext
Powered by