quick question: From which pinot version star-tree...
# general
l
quick question: From which pinot version star-tree index GA is available (production ready)?
k
star-tree index has been GA since a really long time. is there any doc where you found that its not ready?
l
nope Kishore. We are using 0.7.1 and planning to enable star tree index. So, just checking if this feature is production ready in this version 0.7.1
If any major changes, we can upgrade pinot before using this. Have seen some major discussions/jiras in later versions. Hence, just confirming.
k
ah, got it. I think there were few enhancements that allowed dynamic creation and dropping of star tree index
m
BTW, 0.7.1 is quite old, the latest one 0.10.0 was already out a little while ago. You may want to upgrade not for star-tree but for other major features.
l
@Kishore G /@Mayank: just fyi, we upgraded to pinot 0.10.0. Going to try out enabling star-tree index again. Before upgrade, we tried enabling start-tree index and most of our servers went OOM. Are there any specific memory requirements here?
m
Do you have hybrid table? If so, is recommend st index being built outside of Pinot and not on servers.
Also you may want to check your st config (esp maxLeafNodes), to ensure it is optimized for your query set
l
@Mayank: We have hybrid tables but its managed offline flow (minion tasks). Can you please provide/point me to a sample configuration of star tree index. I found some snippets here and there in the docs but couldn't find a full sample anywhere.
If so, is recommend st index being built outside of Pinot and not on servers.
Why is it? Do we have any specific memory/cpu requirement for enabling startree?
m
Depending on star tree config it may be expensive to build. You should only split on columns you filter and group by on. Also what’s the max leaf setting you are using
l
I just enabled using 2 flags available.
Copy code
"enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
m
You can enable it in offline table but may be not in real-time
Also would need to know your data size and query patterns, minion node cpu mem
l
Okay. Will give a try with detailed config.
Can you please provide/point me to a sample configuration of star tree index.
I found some snippets here and there in the docs but couldn't find a full sample anywhere.