hey friends, i have a question regarding

Table Consuming Latency

I have been turning off and on various part of pinot to see how it behaves, this time i decided to turn off for sometime the kafka app that produces the records to pinot, i saw a latency increase when i turned off the app and at least for p99, it was 160ms and now is over a minute, when things like this happen when do you expect pinot to get back to its regular level does it ever get back? I was thinking as the day goes by maybe and this topic start to get less traffic then maybe things come down but I was wondering if that somehow can come back any other way. Ofc this is still pretty fast but I’m wondering what happens if I were to take down the app for a longer time how could that impact the p99 times

Assuming the > 1min latency you are referring to is for consumption, I’d say the consumption catches up pretty fast, however, as you can imagine it is a function of data size, number of partitions, number of servers etc. I’d recommend testing it for practical scenarios you think you will run into.

this is 2 servers 16 partitions the kafka app is only 1 replica and it’s back to speed and we are processing 4k messages/sec

can you share how you measure the "Table Consuming Latency" during the time the app is turned off?

ok so that value is measured by

System.currentTimeMillis() - minConsumingFreshnessMs

in the case your app was turned off. it is basically a linear function of wall-time.

right the app is on already but that metric is not going down

can you make a query?

like MAX time or something like that on the table?

just count* is fine

yea i can

and check if the metrics comes down

they don’t come down

upon checking the code path that metrics only update when a query is being processed.

ohh, this cluster is actively getting some queries

yeah but is that specific for that table?

yes specific for this table

oh. interesting!