I have a everyday load to Pinot tables and I am pl...
# general
s
I have a everyday load to Pinot tables and I am planning to create a segment everyday. Problem is on some days I expect to have very low records so the resulting segment will be too small and will affect performance in longer run. One solution is to do periodic segment merges. Any suggestions?
m
What's your retention? If you have only one segment per day, then there isn't much number of segments to have any severe performance degradation.
s
Yes its 1 segment per day. But I heard from some other people that having smaller segments affected the performance. There might be days (even weeks) when the segment will be in KBs (<10 records)
m
What you want to avoid is too many (say > 1k) smaller segments. There's upcoming feature to merge/rollup small segments.
🙌 1
s
thats cool. I was planning to do a manual consolidate periodically