This message was deleted.
# general
s
This message was deleted.
d
What is your write quorum setting for the cluster? The default is 3, but it is possible to override this at the namespace level. Do you have any rack placement policies on the cluster that might skew the data to one bookie?
e
are you talking about this parameter => managedLedgerDefaultWriteQuorum ? Because in the default values on the chart, the default value is 2
I have this values file =>
Copy code
zookeeper:
    podManagementPolicy: OrderedReady
    securityContext:
      runAsGroup: 0
      runAsUser: 0
    configData:
      PULSAR_MEM: >
        -Xms64m -Xmx128m
      PULSAR_GC: >
        -XX:+UseG1GC
        -XX:MaxGCPauseMillis=10
        -Dcom.sun.management.jmxremote
        -Djute.maxbuffer=10485760
        -XX:+ParallelRefProcEnabled
        -XX:+UnlockExperimentalVMOptions
        -XX:+DoEscapeAnalysis
        -XX:+DisableExplicitGC
        -XX:+PerfDisableSharedMem
  bookkeeper:
    securityContext:
      runAsGroup: 0
      runAsUser: 0
    configData:
      PULSAR_MEM: >
        -Xms128m
        -Xmx1g
        -XX:MaxDirectMemorySize=1g
      PULSAR_GC: >
        -XX:+UseG1GC
        -XX:MaxGCPauseMillis=10
        -XX:+ParallelRefProcEnabled
        -XX:+UnlockExperimentalVMOptions
        -XX:+DoEscapeAnalysis
        -XX:ParallelGCThreads=4
        -XX:ConcGCThreads=4
        -XX:G1NewSizePercent=50
        -XX:+DisableExplicitGC
        -XX:-ResizePLAB
        -XX:+ExitOnOutOfMemoryError
        -XX:+PerfDisableSharedMem
        -verbosegc
      dbStorage_writeCacheMaxSizeMb: "32"
      dbStorage_readAheadCacheMaxSizeMb: "32"
      dbStorage_rocksDB_writeBufferSizeMB: "8"
      dbStorage_rocksDB_blockCacheSize: "8388608"
  broker:
    replicaCount: 4
    configData:
      PULSAR_MEM: >
        -Xms1024m -Xmx1024m -XX:MaxDirectMemorySize=256m
      PULSAR_GC: >
        -XX:+UseG1GC
        -XX:MaxGCPauseMillis=10
        -Dio.netty.leakDetectionLevel=disabled
        -Dio.netty.recycler.linkCapacity=1024
        -XX:+ParallelRefProcEnabled
        -XX:+UnlockExperimentalVMOptions
        -XX:+DoEscapeAnalysis
        -XX:ParallelGCThreads=4
        -XX:ConcGCThreads=4
        -XX:G1NewSizePercent=50
        -XX:+DisableExplicitGC
        -XX:-ResizePLAB
        -XX:+ExitOnOutOfMemoryError
        -XX:+PerfDisableSharedMem
      managedLedgerDefaultEnsembleSize: "2"
      managedLedgerDefaultWriteQuorum: "2"
      managedLedgerDefaultAckQuorum: "2"
      allowAutoTopicCreationType: "partitioned"
  proxy:
    replicaCount: 3
    restartPodsOnConfigMapChange: false
    configData:
      PULSAR_MEM: >
        -Xms64m -Xmx64m -XX:MaxDirectMemorySize=64m
      PULSAR_GC: >
        -XX:+UseG1GC
        -XX:MaxGCPauseMillis=10
        -Dio.netty.leakDetectionLevel=disabled
        -Dio.netty.recycler.linkCapacity=1024
        -XX:+ParallelRefProcEnabled
        -XX:+UnlockExperimentalVMOptions
        -XX:+DoEscapeAnalysis
        -XX:ParallelGCThreads=4
        -XX:ConcGCThreads=4
        -XX:G1NewSizePercent=50
        -XX:+DisableExplicitGC
        -XX:-ResizePLAB
        -XX:+ExitOnOutOfMemoryError
        -XX:+PerfDisableSharedMem
    service:
      type: NodePort
Did i miss something in my configuration ? is the rack placement policies the default data placement ?
d
Yes, those were the settings I was looking for. Thanks for sharing.
I didn’t see the replica count for the Bookies, but based on your original statement, I assume it is 3?
Can you confirm that all bookies are running and that the brokers can communicate with them?
e
It's 4 by default for Bookies on the helm chart
yes all Bookies are running and brokers can communicate with them
d
Is this a single AZ deployment?
e
could it be that the data is only duplicated on one bookie ?
AZ deployment ?
d
When a new segment is created, the bookies that will host the replicas are determined. From that point forward, the data is published directly to those bookies. In the event that a broker can’t write to one of the bookies, the segment is closed, and a new one is created.
AZ = availability zones. Different zones in the same region.
Given the uneven distribution of the data across the bookies, my guess is that there were some network connectivity issues with the bookies that have less data. This would cause the segments they were hosting to be closed prematurely, thus containing less data. This would also result in these bookies not getting selected to host new segments because the brokers couldn’t connect to them.
Even if these connectivity issues were transient and are no longer present, the skewed data distribution would remain. If the data distribution continues to be skewed, e.g., the data volume growth in nominal terms is still skewed to one bookie, then that would indicate that the issue remains.
e
We only have one Pulsar installed on one cluster, so I think it is a AZ deployment
It can be a network connectivity when it is on the same k8s cluster ?
all Bookies are on the same kubernetes cluster
d
So, it sounds like a single AZ deployment. Connectivity issues can be caused by non-networking issues as well, such as pod restarts, OOM conditions, etc.
For now, track the data distribution in the future. If the skew continues, then you have an ongoing issue. If not, it was a transient issue. The good news is that BK should be able to handle unbalanced data distribution just fine.
e
Do you have a simple way to track the data distribution between bookies ?
another important point that if I sum my tenant data on Pulsar Manager i have 266GB x 2 because its duplicated but when i sum my persistence between my bookies pvc it's 990 GB
d
To track the data distribution going forward, I recommend measuring the disk utilization on the
ledgers
directory at the OS level on each bookie. Take the measurement at a fixed interval and compare the rate of growth across the bookies. Each should increase by roughly the same amount. If not, data is still skewing.
IIRC, K8s PVC represents the total capacity you have allocated from the underlying storage class, i.e., “claimed”, and doesn’t represent the total capacity utilized.
e
The numbers I've given refer to the space used on the servers, not to the space "claimed" by the PVC.
d
My mistake.