Hello guys how are you I m having some trouble understanding

Question

Hello guys how are you I m having some trouble understanding the results from the `RealtimeProvisioningHelper` may you help me These are my doubts Why do we need a `numHours` parameter What s the impact of having a consuming segment for a certain amount of time pros cons And what does `Mapped` means in the `Memory used per host` result Is it about the segments in disk This is the results that I got ```RealtimeProvisioningHelper tableConfigFile tmp transaction table json numPartitions 20 pushFrequency null numHosts 4 8 12 16 20 numHours 24 48 72 96 sampleCompletedSegmentDir tmp out transaction 1606528528 1606614928 0 ingestionRate 4 maxUsableHostMemory 16G retentionHours 768 Note Table retention and push frequency ignored for determining retentionHours since it is specified in command See <https docs pinot apache org operators operating pinot tuning realtime> Memory used per host Active Mapped numHosts gt 4 |8 |12 |16 |20 | numHours 24 gt 6 8G 71 9G |3 4G 35 95G |2 72G 28 76G |2 04G 21 57G |1 36G 14 38G | 48 gt 7 33G 72 62G |3 66G 36 31G |2 93G 29 05G |2 2G 21 79G |1 47G 14 52G | 72 gt 8 01G 73 11G |4 01G 36 55G |3 2G 29 24G |2 4G 21 93G |1 6G 14 62G | 96 gt 8 39G 74 08G |4 2G 37 04G |3 36G 29 63G |2 52G 22 22G |1 68G 14 82G | Optimal segment size numHosts gt 4 |8 |12 |16 |20 | numHours 24 gt 20 02M |20 02M |20 02M |20 02M |20 02M | 48 gt 40 04M |40 04M |40 04M |40 04M |40 04M | 72 gt 60 05M |60 05M |60 05M |60 05M |60 05M | 96 gt 80 07M |80 07M |80 07M |80 07M |80 07M | Consuming memory numHosts gt 4 |8 |12 |16 |20 | numHours 24 gt 756 05M |378 02M |302 42M |226 81M |151 21M | 48 gt 1 47G |750 11M |600 09M |450 07M |300 04M | 72 gt 2 15G |1 07G |878 76M |659 07M |439 38M | 96 gt 2 92G |1 46G |1 17G |896 57M |597 71M | Total number of segments queried per host for all partitions numHosts gt 4 |8 |12 |16 |20 | numHours 24 gt 320 |160 |128 |96 |64 | 48 gt 160 |80 |64 |48 |32 | 72 gt 110 |55 |44 |33 |22 | 96 gt 80 |40 |32 |24 |16 |```

Neha Pawar · Answer

Hey < João Comini> thanks for sharing so many details

Neha Pawar · Answer

Have you already gone through this <https docs pinot apache org operators operating pinot tuning realtime realtimeprovisioninghelper>

João Comini · Answer

Yes a lot and my doubts still persist haha

Neha Pawar · Answer

numHours indicates the number of hours the realtime segment will be in CONSUMING state In this state all the ingested data is in memory Periodically based on thresholds the data gets converted to a completed segment and flushed onto disk Now this numHours should be set based on a few factors 1 retention of your kafka stream If your kafka topic retains data for 24h then you don t want to be setting the numHours in Pinot more than 24h If the pinot server gets restarted it has to reconsume everything from the last checkpoint And pinot will rely on the kafka stream to have all that data 2 You want to keep the numHours reasonably low The more the segment consumes the bigger the segment it needs to create and segment creation is memory intensive In case of pinot server restarts the server has to reconsume everything so again resonably low numHours is desired

Neha Pawar · Answer

Typically we don t recommend increasing this more than 24h

João Comini · Answer

Oh right got it What about the number of segments queried If numHours is low i will have a lot more segments right

Neha Pawar · Answer

btw if you re fairly new to the realtime segment concepts this video might help in making sense of some of these terms <https youtu be WoruCQgPhSA>

João Comini · Answer

Thanks I ll take a look

Subbu Subramaniam · Answer

< João Comini> all very good questions Rows are stored in uncompressed format when the segment is consuming but are compressed after it is completed So if you consume for longer time you take in more volatile memory Also like Neha mentioned if you need to restart the server the rows are consumed from the start of the segment again

Neha Pawar · Answer

yes that is true But looks like the max in your case is 320

Subbu Subramaniam · Answer

On the other hand if you set the numhours to be too low then as you pointed out you get too many segments That can be bad for query processing esp in high qps use cases

Neha Pawar · Answer

which is quite small

Subbu Subramaniam · Answer

All the segments still within retention period are in memory mapped as are the consuming segments That is the total mapped memory the active memory is estimated as the most recent 768 hours of data as specified by you in the command line

João Comini · Answer

Nice Thank you guys

João Comini · Answer

Oh i see Right i ll watch Neha s video and see if i get more doubts

Neha Pawar · Answer

that video is only for beginner concepts about realtime consumption and segments Does not cover the provisioning helper but please watch it regardless hah slightly smiling face

Neha Pawar · Answer

Also unrelated if your ingestion rate is only 4 do you really need 20 partitions

Neha Pawar · Answer

the partitioning factor can also help with concerns about too many segments

João Comini · Answer

Yes this one of my concerns The company where i work is huge and we kind of need this solution working quicly The configuration of this topic is not in control of my team so we would need a middleman Flink Heron etc but we don t have that much time hahaha

João Comini · Answer

We could create a simple Java application too that does this work of moving from one topic to another

João Comini · Answer

Oh one more thing by `mapped` you mean that the segments aren t in memory right The segments are in the server s disk and mapped in memory am i missing something

João Comini · Answer

I m asking this because I want to know how much disk space i need to get for my servers

Subbu Subramaniam · Answer

yes by mapped it means that the files in disk are mapped using `mmap`

João Comini · Answer

Right now i feel much more comfortable haha

João Comini · Answer

Last one really i promise and what about resources requests and limits in kubernetes If the active memory + consuming memory use is about 8G how much extra memory would i need to the off heap computations

João Comini · Answer

Should i ask kubernetes for 20G and set XMS and XMX to 16G for safety Just a hypothetical example

Neha Pawar · Answer

good question < Xiang Fu> any recommendations based on this ^^

Xiang Fu · Answer

I would put 4 cpu if you have high qps use case increase this and 32gb ram for request limit and make xmx xmx both to 16g

Neha Pawar · Answer

curious on how you came up with these < Xiang Fu>

João Comini · Answer

Me too haha

Xiang Fu · Answer

that s the t3 2xlarge machine size

Xiang Fu · Answer

8cpu 32gb ram

Xiang Fu · Answer

in general I recommend containers with more ram

Xiang Fu · Answer

so I put the ratio to 4cpu 32gb ram

Xiang Fu · Answer

if you can pick memory optimized sku like r5 xlarge that s the exact fit

João Comini · Answer

hmmm that s something that i ll need to talk about here our nodes are all m5 xlarge

Xiang Fu · Answer

hmm then I would just do 4 cpu 16gb ram for container and set xmx and xms to both 8gb

Xiang Fu · Answer

also I would suggest to use bigger machines like 2xlarge or 4xlarge

João Comini · Answer

All right Thank you so much guys U re awesome heart

João Comini · Answer

I m back hehe And what about offline servers SKUs are there any recommendations What s the priority in this case

Xiang Fu · Answer

Typically we recommend memory optimized SKUs Offline servers typically has less cpu pressure comparing to realtime servers

Xiang Fu · Answer

meanwhile we recommend to enable tag based tier so that Pinot will move persisted segments from realtime servers to offline servers See and

João Comini · Answer

< Rhafik Gonzalez>