https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
p

Prashant Pandey

06/16/2022, 6:48 AM
Hi team, my Pinot servers are consuming from topic but very slowly (one table has a rate of 22 events/s). This is despite high consumer lag in these topics AND the servers’ resources being under-utilised. What might be the reason behind this?
Example log:
Copy code
2022/06/16 06:47:13.991 INFO [LLRealtimeSegmentDataManager_service_call_view_1__5__163__20220616T0632Z] [service_call_view_1__5__163__20220616T0632Z] Consumed 1373 events from (rate:22.707352/s), currentO
Disk util is < 15%
The rate goes up after we restart the server but comes down eventually.
Note that this happened after we upgraded to version 0.10.0. When we rolled-back, the consumption was fine and the lag came down.
Example
The lag went up after moving to 0.10.0. Plummeted once we rolled-back to 0.9.1.
Although this is mostly a red-herring as we’re running 0.10.0 in our other cluster and that is running fine.
k

Kishore G

06/16/2022, 7:26 AM
That’s interesting.
What’s the ingestion rate after you rolled back
p

Prashant Pandey

06/16/2022, 7:31 AM
It’s close to 24k/s for backend_entity_view.
k

Kishore G

06/16/2022, 7:36 AM
Wow.. can’t think of anything that would cause this issue
p

Prashant Pandey

06/16/2022, 7:58 AM
Yes actually I am gonna redeploy 0.10.0 and see if it goes down again.
And report back
l

Laxman Ch

07/29/2022, 8:01 AM
This is just a monitoring issue. Not a problem actually. Set
group.id
in table streams config. Fixes the issue