Channels
#troubleshooting
Title
p
Pradeep
09/30/2020, 12:12 AMshouldn;t the segment rotate if 2h is hit?
n
Neha Pawar
09/30/2020, 12:17 AM
yes it should have flushed after 2 hours.
-was this working fine and then suddenly you’re seeing this behavior?
-was the server being restarted frequently? 2h calculation will get reset if restarts happen
-any exceptions in server/controller logs about segment completion?
-is this happening for all partitions?
p
Pradeep
09/30/2020, 12:27 AMlooks like it’s happening on partitions part of one of the server. I see some
OutOfMemoryErrorscould that be the reason?
n
Neha Pawar
09/30/2020, 12:29 AM
if OOM is causing the server to crash/ restart, then yes that could be the reason
is the 500M segment size realistic? do you end up creating that big segments?
p
Pradeep
09/30/2020, 12:29 AMoh, I don’t see any network activity
so consumers sort of got killed
yeah I could decrease that size
ideally killing the entire server would have been better
since we didn’t notice this
n
Neha Pawar
09/30/2020, 12:30 AM
set it closer to what you’re actually wanting to generate (or what the 2h is generating). That way the segment build will be triggered based on the size
oh so consumers got killed but server is alive?
p
Pradeep
09/30/2020, 12:31 AMI think in this case consumers just went down and never came up, are there any metrics/checks that are there to expose it
yeah seems so
n
Neha Pawar
09/30/2020, 12:31 AM
Copy code
LLC_PARTITION_CONSUMINGp
Pradeep
09/30/2020, 12:32 AMdoes that represent number of consuming segments?
let me check that
n
Neha Pawar
09/30/2020, 12:34 AM
i believe there will be one per consumer, and it should be 1 all the time
s
Subbu Subramaniam
09/30/2020, 4:08 PM@Neha Pawar the 2h will not get reset if servers get restarted. We take the start time based on segment creation time, so if it is restarted within an hour, the segment will attempt to consume for roughly an hour more.
@Pradeep you should really run the realtime provisioning helper (you can use the segments you have already created) to decide what parameters to set for consumptoon. See https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime
p
Pradeep
09/30/2020, 4:12 PMWe got the memory under control using offheap, but yeah I will try the provisioning helper and update our config. This OOM was result of an expensive query we ran, that seemed to have just killed the consumers while the server was up IIUC.
s
Subbu Subramaniam
09/30/2020, 4:41 PM2h and 500M seems way off. Hopefully the tool will help you configure better values with the right tradeoff that you can afford
p
Pradeep
09/30/2020, 5:41 PMcurious why 2h and 500M seems off? we see order of 400M segment sizes
s
Subbu Subramaniam
09/30/2020, 5:53 PMThere are multiple variables that include the number of partitions in your stream, the "width" of each row (how many bytes they occupy in mem), and the rate of ingestion per partition, the number of hosts you have consuming, the number of replicas, memory in each host, etc. If you have already accounted for all this, and set it to 2h/500M then that is fine. In fact, if you are producing 400M segments (assuming without compression), you are probably in the right ballbark and the tool may recommend the same.
p
Pradeep
09/30/2020, 5:55 PMgot it, thanks