https://pinot.apache.org/ logo
p

Pradeep

09/30/2020, 12:12 AM
shouldn;t the segment rotate if 2h is hit?
n

Neha Pawar

09/30/2020, 12:17 AM
yes it should have flushed after 2 hours. -was this working fine and then suddenly you’re seeing this behavior? -was the server being restarted frequently? 2h calculation will get reset if restarts happen -any exceptions in server/controller logs about segment completion? -is this happening for all partitions?
p

Pradeep

09/30/2020, 12:27 AM
looks like it’s happening on partitions part of one of the server. I see some
OutOfMemoryErrors
corresponding to query execution
could that be the reason?
n

Neha Pawar

09/30/2020, 12:29 AM
if OOM is causing the server to crash/ restart, then yes that could be the reason
is the 500M segment size realistic? do you end up creating that big segments?
p

Pradeep

09/30/2020, 12:29 AM
oh, I don’t see any network activity
so consumers sort of got killed
yeah I could decrease that size
ideally killing the entire server would have been better
since we didn’t notice this
n

Neha Pawar

09/30/2020, 12:30 AM
set it closer to what you’re actually wanting to generate (or what the 2h is generating). That way the segment build will be triggered based on the size
oh so consumers got killed but server is alive?
p

Pradeep

09/30/2020, 12:31 AM
I think in this case consumers just went down and never came up, are there any metrics/checks that are there to expose it
yeah seems so
n

Neha Pawar

09/30/2020, 12:31 AM
Copy code
LLC_PARTITION_CONSUMING
p

Pradeep

09/30/2020, 12:32 AM
does that represent number of consuming segments?
let me check that
n

Neha Pawar

09/30/2020, 12:34 AM
i believe there will be one per consumer, and it should be 1 all the time
s

Subbu Subramaniam

09/30/2020, 4:08 PM
@Neha Pawar the 2h will not get reset if servers get restarted. We take the start time based on segment creation time, so if it is restarted within an hour, the segment will attempt to consume for roughly an hour more.
@Pradeep you should really run the realtime provisioning helper (you can use the segments you have already created) to decide what parameters to set for consumptoon. See https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime
p

Pradeep

09/30/2020, 4:12 PM
We got the memory under control using offheap, but yeah I will try the provisioning helper and update our config. This OOM was result of an expensive query we ran, that seemed to have just killed the consumers while the server was up IIUC.
s

Subbu Subramaniam

09/30/2020, 4:41 PM
2h and 500M seems way off. Hopefully the tool will help you configure better values with the right tradeoff that you can afford
p

Pradeep

09/30/2020, 5:41 PM
curious why 2h and 500M seems off? we see order of 400M segment sizes
s

Subbu Subramaniam

09/30/2020, 5:53 PM
There are multiple variables that include the number of partitions in your stream, the "width" of each row (how many bytes they occupy in mem), and the rate of ingestion per partition, the number of hosts you have consuming, the number of replicas, memory in each host, etc. If you have already accounted for all this, and set it to 2h/500M then that is fine. In fact, if you are producing 400M segments (assuming without compression), you are probably in the right ballbark and the tool may recommend the same.
p

Pradeep

09/30/2020, 5:55 PM
got it, thanks