Hey Pinot team 👋 Slack is recently using Pinot upsert to build the realtime table Could you please you share the recommended metrics to evaluate if the realtime cluster is in the healthy status thanks Based on what we know, heap utilization could be a critical metrics to validate is the node is overwhelming and we can confirm that our heap utilization is under 80% based on this calculation

jvm_memory_bytes_used{instance=~'.*staging.*',area="heap"} / on(instance) group_left jvm_memory_bytes_max{instance=~"pinot-.*",instance=~".*staging-.*",instance=~".*realtime-.*",area="heap"} * 100

cc: @Neeraja Sridharan @Sai Tarun Tadakamalla

Number of PKs is also something to watch for. For heap you want to check the min heap occupied over a period of time

@Mayank thank you for staging in here 🙇 here is our PK dashboard for the past 24h and 7d; I guess we need the specific function to calculate the storage it takes?

Generally speaking, we have seen max of 100M PK per server (across all tables) for small key sizes. A better way is to monitor the min value of heap over say 1 hr. That tells you the old gen occupied. And for a healthy server you want it < 30% of xmx

we have seen max of 100M PK per server (across all tables) for small key sizes

Thanks for pointing this out! I just refine the dashboard to reflect PK/server; looks like the max PK/server is something like 16M; so we should be safe with that 100M PK in this aspect

Note, it depends on the size of each PK. 100M is for the smaller size (like int or long). Also, treat that number as the absolute upper bound, it might be much lower in your case.

We just also build up

Min heap over 1hr per server (Old Gen occupied)

with

min_over_time(
  jvm_memory_bytes_used{
    instance=~"pinot-realtime-server-staging-prod.*",
    area="heap"
  }[1h]
)
/
jvm_memory_bytes_max{
  instance=~"pinot-realtime-server-staging-prod.*",
  area="heap"
} * 100

like you suggested; looks like some nodes can spike to 40%

Heap and GC are good metrics for upsert tables to make sure resource are not exhausted.

@Ting Chen thanks for the suggestion; we did some tuning especially

-XX:InitiatingHeapOccupancyPercent=25 -XX:ConcGCThreads=4 -XX:-G1UseAdaptiveIHOP

control the total heap usage from ~90% to <70%

Min heap doesn't necessarily need to be < 30% if your max is contained within < 70%. These are just soft guidelines.

@Mayank good to know, thanks for sharing this!

Is

-XX:InitiatingHeapOccupancyPercent=25 -XX:ConcGCThreads=4 -XX:-G1UseAdaptiveIHOP

recommended as well?

> Is

-XX:InitiatingHeapOccupancyPercent=25 -XX:ConcGCThreads=4 -XX:-G1UseAdaptiveIHOP

recommended as well? Slack's pinot cluster leverages this config to control heap max < 70%; @Mayank @Ting Chendo you see any risk with this config?