questions; are the servers supposed to have as many cores as partitions on a kafka topic (?), if that’s the case what’s the best way to scale up pinot setups, the more tables we add the more the servers will have to ingest and we will get into resource contention given the cores on a server, how do you all manage this?

No, that is not a requirement. For scaling up, you’ll have to check resource usage based on your workload to see if scaling is needed

we have observed for our setup at least that the more we have been adding tables we have been scaling up our servers at least with more cores

Yeah that is a function of workload, and not a requirement on number of partitions vs number of cores

also number of partitions is also not a function of number of server nodes right? like if i have 16 partitions in my kafka topic it doesn’t mean i need 16 servers

Yeah, no such requirement. It is a function of total ingestion across partitions, while serving queries and perhaps even generating segments at peak (aka workload). You can refer to this to part blog https://www.startree.ai/blog/capacity-planning-in-apache-pinot-part-1

one thing that we do see is that for example at our size tables can get impacted

You can use the ingestion rate limiter to limit the max rate at which one table can ingest, so that it doesn’t steal resources from other tables when bootstrapping