shouldn;t the segment rotate if 2h is hit?

yes it should have flushed after 2 hours. -was this working fine and then suddenly you’re seeing this behavior? -was the server being restarted frequently? 2h calculation will get reset if restarts happen -any exceptions in server/controller logs about segment completion? -is this happening for all partitions?

looks like it’s happening on partitions part of one of the server. I see some

OutOfMemoryErrors

corresponding to query execution

if OOM is causing the server to crash/ restart, then yes that could be the reason

oh, I don’t see any network activity

set it closer to what you’re actually wanting to generate (or what the 2h is generating). That way the segment build will be triggered based on the size

I think in this case consumers just went down and never came up, are there any metrics/checks that are there to expose it

does that represent number of consuming segments?

i believe there will be one per consumer, and it should be 1 all the time

@Neha Pawar the 2h will not get reset if servers get restarted. We take the start time based on segment creation time, so if it is restarted within an hour, the segment will attempt to consume for roughly an hour more.

We got the memory under control using offheap, but yeah I will try the provisioning helper and update our config. This OOM was result of an expensive query we ran, that seemed to have just killed the consumers while the server was up IIUC.

2h and 500M seems way off. Hopefully the tool will help you configure better values with the right tradeoff that you can afford

curious why 2h and 500M seems off? we see order of 400M segment sizes

There are multiple variables that include the number of partitions in your stream, the "width" of each row (how many bytes they occupy in mem), and the rate of ingestion per partition, the number of hosts you have consuming, the number of replicas, memory in each host, etc. If you have already accounted for all this, and set it to 2h/500M then that is fine. In fact, if you are producing 400M segments (assuming without compression), you are probably in the right ballbark and the tool may recommend the same.

got it, thanks