Hi team, I am trying to Run the rebalance and it s...
# troubleshooting
v
Hi team, I am trying to Run the rebalance and it says table is already balanced. however the number of segments is uneven on servers. We have realtime table with upserts enabled, pinot 1.2.
Copy code
Server_masked105.[masked]_8098	420
Server_masked106.[masked]_8098	420
Server_masked107.[masked]_8098	414
Server_masked101.[masked]_8098	414
Server_masked101.[masked]_8098	412
Server_masked102.[masked]_8098	412
Server_masked103.[masked]_8098	410
Server_masked104.[masked]_8098	410
Server_masked108.[masked]_8098	207
Server_masked109.[masked]_8098	207
Server_masked110.[masked]_8098	207
Server_masked111.[masked]_8098	207

Rebalance fails with 
 "status": "NO_OP",
  "description": "Table is already balanced",
Copy code
{
  "jobId": "77e13819-a66e-4612-a4ef-15485da4f4b3",
  "status": "NO_OP",
  "description": "Table is already balanced",
  "instanceAssignment": {
    "CONSUMING": {
      "instancePartitionsName": "test_counters_CONSUMING",
      "partitionToInstancesMap": {
        "0_0": [
          "Server_masked104.[masked]_8098",
          "Server_masked105.[masked]_8098",
          "Server_masked106.[masked]_8098",
          "Server_masked107.[masked]_8098",
          "Server_masked101.[masked]_8098",
          "Server_masked102.[masked]_8098",
          "Server_masked103.[masked]_8098",
          "Server_masked104.[masked]_8098",
          "Server_masked105.[masked]_8098",
          "Server_masked106.[masked]_8098",
          "Server_masked107.[masked]_8098",
          "Server_masked108.[masked]_8098",
          "Server_masked109.[masked]_8098",
          "Server_masked110.[masked]_8098",
          "Server_masked111.[masked]_8098",
          "Server_masked112.[masked]_8098",
          "Server_masked113.[masked]_8098",
          "Server_masked114.[masked]_8098",
          "Server_masked115.[masked]_8098",
          "Server_masked116.[masked]_8098",
          "Server_masked117.[masked]_8098",
          "Server_masked118.[masked]_8098",
          "Server_masked119.[masked]_8098",
          "Server_masked120.[masked]_8098",
          "Server_masked121.[masked]_8098",
          "Server_masked122.[masked]_8098",
          "Server_masked123.[masked]_8098",
          "Server_masked124.[masked]_8098",
          "Server_masked125.[masked]_8098",
          "Server_masked101.[masked]_8098",
          "Server_masked102.[masked]_8098",
          "Server_masked103.[masked]_8098"
        ]
      }
    }
  },
  "segmentAssignment": {
    "test_counters__0__0__20250411T1059Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__100__20250415T2304Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__101__20250416T0004Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__102__20250416T0104Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__103__20250416T0204Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__104__20250416T0304Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__105__20250416T0404Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__106__20250416T0504Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__107__20250416T0604Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__108__20250416T0704Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__109__20250416T0804Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__10__20250411T2100Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__110__20250416T0904Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__111__20250416T1004Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__112__20250416T1104Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__113__20250416T1205Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__114__20250416T1305Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__115__20250416T1405Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__116__20250416T1505Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__117__20250416T1605Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__118__20250416T1705Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__119__20250416T1805Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__11__20250411T2200Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__120__20250416T1906Z": {
      "Server_masked104.[masked]_8098": "ONLINE",
      "Server_masked105.[masked]_8098": "ONLINE"
    },
    "test_counters__0__121__20250416T2006Z":
x
pinot has to keep one entire partition on the same server.
so maybe the data are skewed?
v
yes but these 6-7 pinot server shave 2x the number of segments wrt to other servers. We have total 32 pinot servers
x
how many kafka partitions you have for this
test_counters
table?
if this table only has one partition then all the segments will be on server 104 and 105
v
test_counters has 20 partitions. the above output is truncated as it has huge list
just curious, how pinot decide where to place the consuming segment because for another table i see 8 servers have 700 segments and rest 24 server has 350 segments and rebalance doesnt run says table is already balanced. also rebalance tenant also show tables are balanced
x
so one kafka partition is served by 2 pinot servers as you configured 2 replicas, this data won’t be spread across all the servers
if for this table, it will be 40 partitions across 32 servers, then 8 servers will have 2 partitions, and 24 servers will have 1 partition
then we consider this table is balanced
it’s not about number of segments per server
so you will see 8 servers have double segments
v
okay understood. thanks alot 🙏 one followup question - Is there any Pinot config or balancer logic we can apply periodically to auto-heal such imbalance for consuming segments specifically?
x
but it’s balanced right?
the realtime partition data has to be served from the same server
if you want to rebalance all the persisted data, what you really need is a hybrid table and use segment relocator to move segments from realtime table to offline table then you can archieve this kind of fully balance
v
yeah my concern is regarding resource pressure on the servers with 2x segments. like if a server has more consuming segments, could that result in higher heap and disk usage over time
you can add 8 more servers to balance this 😉
the resource consumption for consuming segments is fixed, you will have 8 servers doubled the resources due to consumption
v
got it. So, would it help if the Kafka topic partition count is aligned more closely with the number of Pinot servers or a multiple of it?
x
yes
yes, you can increase kafka partitions from 20 to 32, it will balance all the workload
v
awesome, very helpful @Xiang Fu thanks