Hey Folks*.* Our team is currently investigating the feasibility of StarRocks at our org and carrying out large scale testing. We want StarRocks to be able to scale up to be able to service atleast 800 queries per second with under 1 second latency without any latency performance degradation. In our testing we noticed we can definitely get there if we provide it enough static compute. Great. But if you want dynamic compute and scale via HPA so you are not spending a ton of $$$ at all times there are issues. Issue is, we notice the latency of queries increasing at the exact moment we add a CN node during a scale out operation or drop a CN node during a scale in. Maintaining 1 second latency during scale in and scale out operations is mission critical for us to utilize StarRocks. Some queries even fail and have to be rerun which is not necessarily bad as long as they fail fast although the failures are obviously not preferred. We are planning on using StarRocks for dashboards so 1 second latency is imperative. We are using the official Helm Chart and version 3.5.15 on EKS and have tried playing with a bunch of parameters Observations: • Currently some running queries fail while successful queries experience increased latency when under high load upon scaling up CN nodes. It seems due to tablet reassignments based on what we see in the logs are causing this • We notice the increased latency everytime a CN Node is added • We notice the failures happens only after we have added a few more CN nodes (do not happen on the first few CN nodes that we add). This might be also simply because we are steadily increase the load as we go further in the scale test • We are able to prevent errors by disabling tablet rebalancing but obviously then its pointless to scale because the new CNs are not taking on any work. • We have also tried reducing how quickly rebalancing occurs on the new CNS but our benchmarking StarRocks cluster does not have many tablets cause we only imported a sample set of our data, so we are only working with 600 tablets. Questions: • Can someone please confirm the increased latency due to tablet rebalancing on new CNs is a known issue with the product? And if so are there any recommendations on we mitigate it to accomplish our goal of always have 1 second latency? • Same question for scaling in CN nodes when load has subsided. Any recommendations people can offer are much appreciated.