https://pinot.apache.org/ logo
Join Slack
Powered by
# pinot-realtime-table-rebalance
  • j
    Seems the problem is that there are 12 kafka partitions (streaming partition * replication), but only 9 servers
  • j
    So 3 servers will have 2 partitions to consume, the other 6 has 1 partition
  • t
    yes. so the 9 servers' perf is in fact worse than 6 servers.
  • y
    no. this topic has 4 partitions
  • y
    but 3 repication factor
  • t
    I also observed the document distribution is not even -- the latter 3 servers has 50% more doc than the original 6.
  • t
    is that expected?
  • j
    The difference between LLC and realtime is that all the segments for one partition will be hosted on the same server, so think of partition as the smallest unit of the table
    n
    • 2
    • 2
  • t
    what do you mean by the diff between LLC and realtime? I thought LLC is realtime?
  • j
    Sorry, LLC and offline
  • j
    Performance wise, 9 servers should be similar to 6 servers because the server load on the 3 new servers are the same as before
  • j
    Do you use partitioning or replica-group routing for this table?
  • y
    we use default
  • t
    we want to use replica-group routing for this tenant (right now it has 12 servers)
  • t
    otherwise each query got fanned out to all 12 now -- which could depend on the slowest server.
  • j
    For LLC table, because of the nature of the streaming partition, the segments are already assigned into replica-groups
  • t
    Copy code
    {
 "tableName": "pinotTable", 
 "tableType": "REALTIME",
 "routing": {
 "instanceSelectorType": "replicaGroup"
 }
 ..
}
  • j
    Simple enable replica-group routing should do the work
  • j
    Yes, correct
  • t
    so we just need to add the above and restart broker?
  • j
    Let me check, I think we have an API to avoid restarting broker
  • j
    You can use the broker rebuild routing API to enable it:
    Copy code
    @PUT
@Produces(MediaType.TEXT_PLAIN)
@Path("/routing/{tableName}")
  • j
    (table name here is the full table name, e.g. 
    pinotTable_REALTIME
    )
  • n
    though this won’t solve you issue of some servers seeing twice the load. 1 server in each replica group is still going to have the same behavior
  • j
    I think they already scaled up the cluster to 12 servers
  • n
    o okay
  • t
    yes.
  • t
    looks like 6-9 was not a good idea. 6->12 is.
  • y
    Thanks @User @User for the help
  • r
    Hi everyone... I am using realtime ingestion. Kafka source has a retention period of 3 days, but the lag in Pinot said 4-5 days and was increasing. When I restated the servers the lag dropped to 2-2.5 days and reduced quite drastically and overnight dissipated. Can you pls help me debug this?