https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team I am planning to add some invertedColumn i...
# general
a
Hi team I am planning to add some invertedColumn indices and bloomFilter Config in a Pinot Realtime upsert table with 2 server partitions I wanted to confirm whether the indices are created offheap or in heap.I know for Consuming segment it is inside heap.Can someone share a document where it is specified ?
m
Inv index for consuming segments is on heap, rest is all off heap.
a
Thanks @Mayank is there a metric which can tell directly how much off heap memory will be consumed ?
m
All segments are memory mapped, so whatever size you have. But if you are asking for used memory, there should be system level metrics for that
a
Ok what about the index specifically I am using 0.12.0
m
Inv index only? I don’t think there is a metric. But you can check index size from api
But how are you using this info?
a
Which API ..? We are going to add indices in Pinot prod.But due to our cluster limitations I am not able to add more RAM so I wanted an idea how much RAM memory will be consumed by all indices..I don't specifically want inverted indices
m
Memory map means you don’t need that much RAM as size of data or index
Pages are fetched in on demand and flushed out
a
Yeah I understand that although we now have 30GB of segment on a partition and only 6 GB of RAM to spare which is why I wanted to know Although adding indices will only make things better
m
What is the active page cache size
a
Is there a metric for it or this system level ?
m
If you are using k8s that should be available
a
@Mayank I have found the below numbers on 2 of our partitions.I apologize if I have found the wrong data
Copy code
Partition with Segments size 22.23GiB
free -h
               total        used        free      shared  buff/cache   available
Mem:            30Gi        23Gi       419Mi       3.0Mi       6.9Gi       6.9Gi
Swap:             0B          0B          0B

cat /proc/meminfo | grep -i cache
Cached:          6859584 kB
SwapCached:            0 kB

========================================================================================================================================

Partition with Segments size 36GiB
free -h
               total        used        free      shared  buff/cache   available
Mem:            30Gi        24Gi       401Mi       3.0Mi       5.8Gi       5.8Gi
Swap:             0B          0B          0B

cat /proc/meminfo | grep -i cache
Cached:          5714572 kB
SwapCached:            0 kB
@Mayank I added indexes last night and I have started seeing some of my queries starting to timeout due to high ResponseDelayMs
Copy code
serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs,RequestSentDelayMs);cdp-dl-pinot-k8s-server-4_R=0,16627,3155131,1,0;cdp-dl-pinot-k8s-server-6_R=0,3,790,0,1,offlineThreadCpuTimeNs(total/thread/sysActivity/resSer):0/0/0/0,realtimeThreadCpuTimeNs(total/thread/sysActivity/resSer)
These are some of the filter params.Do you know what could be the problem
Copy code
"exceptions": [],
  "numServersQueried": 2,
  "numServersResponded": 2,
  "numSegmentsQueried": 1482,
  "numSegmentsProcessed": 262,
  "numSegmentsMatched": 247,
  "numConsumingSegmentsQueried": 2,
  "numConsumingSegmentsProcessed": 0,
  "numConsumingSegmentsMatched": 0,
  "numDocsScanned": 56587,
  "numEntriesScannedInFilter": 342097,
  "numEntriesScannedPostFilter": 282935,
  "numGroupsLimitReached": false,
  "totalDocs": 256775252,
  "timeUsedMs": 16633,
  "offlineThreadCpuTimeNs": 0,
  "realtimeThreadCpuTimeNs": 0,
  "offlineSystemActivitiesCpuTimeNs": 0,
  "realtimeSystemActivitiesCpuTimeNs": 0,
  "offlineResponseSerializationCpuTimeNs": 0,
  "realtimeResponseSerializationCpuTimeNs": 0,
  "offlineTotalCpuTimeNs": 0,
  "realtimeTotalCpuTimeNs": 0,
  "segmentStatistics": [],
  "traceInfo": {},
  "minConsumingFreshnessTimeMs": 1702588549533,
  "numSegmentsPrunedByBroker": 0,
  "numSegmentsPrunedByServer": 1220,
  "numSegmentsPrunedInvalid": 0,
  "numSegmentsPrunedByLimit": 0,
  "numSegmentsPrunedByValue": 1220,
  "explainPlanNumEmptyFilterSegments": 0,
  "explainPlanNumMatchAllFilterSegments": 0,
  "numRowsResultSet": 40
m
Is there a memory usage increase after adding these indexes?
a
You mean heap memory or is there a specific metric.I have not seen any specific memory increase
Open in Slack
PreviousNext
Powered by