Hello, our team is interested in exploring realtim...
# general
a
Hello, our team is interested in exploring realtime upsert tables, but I am curious about the maintenance and scaling of the table going forward. Conceptually I understand that the same partition of data has to always route through the same server for this to work. Referencing these docs: https://docs.pinot.apache.org/manage-data/data-import/upsert-and-dedup/upsert#enable-upserts-in-pinot I see we must use
Copy code
{
  "routing": {
    "instanceSelectorType": "strictReplicaGroup"
  }
}
and also
numInstancesPerPartition=1
With this is mind, is it still possible to scale up the number of servers horizontally once the table is set up? And what is the process then to do so?
Let's say we have a kafka topic with 48 partitions. We start with 4 realtime servers for the upsert table, so presumably each is working with 12 partitions. If we see that traffic is increasing and we need to scale horizontally, is it possible to increase our server count to 8 servers? Does this require pausing ingestion and rebalancing to do so? Is it possible at all?
m
You can safely scale the number of servers. The only caveat is scaling number of partitions. So you want to slightly over provision partition count in upstream
👍 1
a
Okay thanks that's good to know