I see this error.

fyi, we can look into this tomorrow, not that important at the moment

cool, sure we can debug it tmr

Hi everyone, how do you monitor the Pinot server memory and GC metrics usually? Are there any shareable Grafana dashboards in the community?

Hi, we wanted to deploy a version of pinot that has the deepstore bypass and peer download fix. Are these the pr's we need to patch 0.4.0 with? [Deepstore by-pass]Add a Deepstore bypass integration test with minor bug fixes. #5857 by chenboat was merged 7 days ago [Deepstore by-passing]Introduce a subclasses SplitSegmentCommitter which will proceeds to commit even if the segment upload fails. #5700 by chenboat was merged on Jul 23 Refactor how Pinot controller stores segment download url in Zookeeper to deal with peer uri format #5639 by chenboat was merged on Jul 2 Add a new table config field for peer segment download. #5478 by chenboat was merged on Jun 2 [Part 4] Deep-store bypass for LLC: Add a peer to peer segment fetcher. #5336 by chenboat was merged on Jul 2 [Part 3.1] Deepstore by-pass: Add a new best effort segment uploader with bounded upload time and d… #5314 by chenboat was merged on May 5 [Part 2] Deepstore by-pass in LLC: introduce segment uploader for segment split commit. #5277 by chenboat was merged on Apr 24 Add access control for Pinot server segment download api. #5260 by chenboat was merged on Apr 20 Add a new server api for download of segments. #5221 by chenboat was merged on Apr 15 cc: @Onha Choe

I have about 2k segments in ERROR state out of a total of 10k segments. It is because the data in controller was missing while i triggered rebalancing. I have since pushed the missing data to the controller. now what is the best way to fix the ERROR segments ?

@Yash Agarwal You might need to restart the servers

Another approach is to disable then reenable the table to trigger the segment reload. But I feel it is safer to restart the servers to clean up the unexpected states

how do i troubleshoot this?

Question: If I add a sort inverted index on a column, what happens to the timestamp column? is the data sorted by the new column only, ignoring the timestamp or is it sorted by both? Not that clear from the docs at https://docs.pinot.apache.org/basics/indexing/inverted-index#sorted-inverted-index

what does add sort inverted index mean

let's say i'm adding sort inverted index on

column1

I think our doc is a bit confusing on

sorted inverted index

.

There's only one indexing file for sorted columns that duals as sorted as well as inverted index.

It is created if the column in input data is sorted.

If you sort your input data on a column, it might (and will likely) unsort the timestamp.

That makes sense. Thanks for clarifying.

Has anyone seen this? What's the best way to add STAR tree indexing and verify that?

Will distintCountBitmap and distinctCount return the same result on Integer columns?

@Yash Agarwal Yes, they will always be the same

@Mayank ^^

Hi, QQ I noticed our realtime pinot servers memory-usage and cpu usages keeps growing with time Once I restart the realtime servers memory usage goes back to a baseline and keeps growing again This doesn’t happen with offline servers, makes me wonder if there is a memory leak somewhere or maybe buffers are getting recycled in realtime servers? Wondering if anybody noticed this kind of behavior?

Most likely this is yuor consuming segments processing queries. Consuming segments store raw (uncompressed) dayta in memory, and persist every once in a while depending on segment size setting,

See https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime

@Pradeep ^^

You may need to introduce some inverted indices to process queries faster, so that your cpu usage does not increase. Memory usage is indeed expected to increase, but you can control it by settings described in the page abive

I was assuming that once the segments get rolled over, memory usage should restore? (memory kept increasing over couple of days, where as segment roll over time period is much less than that)

I will try the offheap allocation

thanks

Do you have the memory graph over multiple days?