Are you looking at JVM graph or system memory graph

what is the retention of your table? By default, completed segments are kept on the same servers that they are consumed in Is this a hybrid or realtime-only table? It will be useful to run the

RealtimeProvisioningHelper

to get an idea of memory usage

This is the memory usage graph, I am looking at system memory (no other active processes live on the system)

retention is set to more than 30 days I believe, it’s a. hybrid table

I can try running that

If it is hybrid table, with a frequent push on the offline side, your retention for realime table should be short. e.g, 5 days for daily offline push

thats expected and is the right thing, OS is pretty good at managing the system memory

I did observe query latencies going on when the system memory is high, I believe if I use the offheap memory then page caches should cleared up, once the mmaped memory used for backing realtime segments are deleted. Let me try that and see (don’t want to touch the system now, will try this in offpeak hours)

can you do lsof on the realtime process

Is there a way to configure which servers should be part of a single replica group? or will pinot randomly assign them ?

it should be randomly

@Yash Agarwal you should be able to create a znode with specific assignments of each replica group if desired. @Jackie have we documented this?

@Neha Pawar wondering if you know what’s going on here, Jackie refered to you I have segment which is in

consuming

state for close to 20h, from zk metadata

"segment.creation.time": "1601350869326",

But my table config has segment rolling config as

"realtime.segment.flush.threshold.size": "0",
        "realtime.segment.flush.threshold.time": "2h",
        "realtime.segment.flush.desired.size": "500M",

Just a note on the star-tree documentation, supported functions list DISTINCT_COUNT_HLL but it seems like correct way to specify that is DISTINCTCOUNTHLL__<colname>

@Pradeep Good point, we should also support

DISTINCT_COUNT_HLL__<colname>

I tried that but it threw an exception

Yeah, will submit a fix for that

thanks

@Pradeep Here is the fix: https://github.com/apache/incubator-pinot/pull/6079. Once it's merged, it should accept both format

@Chinmay Soman ^^

Hi, I am trying to optimize a query of the format

select colA, distinctCountHll(colb)
from table
where timestamp > X
group by colA

We added a star-tree with dimensionsSplitOrder: [“colA”] and aggregateFunction as DistinctCountHLL__colB I am not seeing much query-time improvements, I am doing a comparison against aggregate by colC which is not part of star-treee and see very similar times. I see that star-tree index is getting generated. Wondering if I am missing someething?

@Pradeep output stats?

timeMs=8541,docs=361440597/16479407613,entries=336003488/722881194,segments(queried/processed/matched/consuming/unavailable):5229/227/227/8/0,consumingFreshnessTimeMs=1601574112264,servers=4/4,groupLimitReached=false,exceptions=0,serverStats=(Server=SubmitDelayMs,ResponseDelayMs,ResponseSize,DeserializationTimeMs);172.31.17.90_R=1,8536,68556,0;172.31.30.139_O=1,3,372,0;172.31.34.149_O=1,4,372,0;172.31.24.127_R=1,8470,69174,0,

2servres have old data, so thy don’t match anything

its still scanning a lot

yaeh

@Jackie ^^

Also I think timestamp should be part of the dimensionsSplitOrder right ?

Yes, yash gives the answer

Can you try removing the filter on

timestamp

and see the latency?