https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, my Pinot servers are consuming from topic...
# troubleshooting
p
Hi team, my Pinot servers are consuming from topic but very slowly (one table has a rate of 22 events/s). This is despite high consumer lag in these topics AND the servers’ resources being under-utilised. What might be the reason behind this?
Example log:
Copy code
2022/06/16 06:47:13.991 INFO [LLRealtimeSegmentDataManager_service_call_view_1__5__163__20220616T0632Z] [service_call_view_1__5__163__20220616T0632Z] Consumed 1373 events from (rate:22.707352/s), currentO
Disk util is < 15%
The rate goes up after we restart the server but comes down eventually.
Note that this happened after we upgraded to version 0.10.0. When we rolled-back, the consumption was fine and the lag came down.
Example
The lag went up after moving to 0.10.0. Plummeted once we rolled-back to 0.9.1.
Although this is mostly a red-herring as we’re running 0.10.0 in our other cluster and that is running fine.
k
That’s interesting.
What’s the ingestion rate after you rolled back
p
It’s close to 24k/s for backend_entity_view.
k
Wow.. can’t think of anything that would cause this issue
p
Yes actually I am gonna redeploy 0.10.0 and see if it goes down again.
And report back
l
This is just a monitoring issue. Not a problem actually. Set 
group.id
in table streams config. Fixes the issue
Open in Slack
PreviousNext
Powered by