@User what do you mean?

I meant we dont allow generating star-tree index on the fly. There is no reason to do this other than Star-tree index generation is a bit resource intensive and might impact query performance while index is being generated

yep, that is what I though. I'm trying to figure out automation that will be required to do those updates, and currently I'm thinking that if we need to update star tree we will run a Segment generation job (Spark or Flink based one) and upload those new segments into Pinot cluster. Does it sound like a ok idea?

yes^^

great! So, which one is a better idea -> use some external dataset to do it (hadoop) or use existent Pinot segment files?

There is something called as Minion

It can be used for doing these tasks.. it’s used for running GDPR related tasks or optimize segments etc

Take a look at that. You can add startree index gen task to it

I thought minions are deprecated, is it a good idea to use them?

Yes. We will add more and more tasks to minion

Curious, where you got that idea about minions being deprecated. If it’s a documentation issue, we should fix that.

nope, I think somebody mentioned it at some point

ok, if we use minions -> what is the right way to use them on kubernetes? Just have a cronjob that launches a specific task? Any good examples on github we can check?

Minion uses Helix Task Framework. @User can you share the design doc and some examples for Minion

so, minions always run?

Yes

what is the trigger to execute the work?

IIRC, the tasks are defined using Helix task framework

the trigger can be manual or scheduled

or it can also be based on another resource

for e.g. you can configure it to run some task whenever a new segment is uploaded

oh, that is good to have

or you can run something every day at 12.00 mid night

or you can even write your own task generator

the good thing about minion framework is it abstracts out all the common things needed to perform some action on a segment

it can download segment, upload segment, do the bookkeeping etc and also provide pinot segment readers etc

nice. will need to dig into this. On kube it feels like a waste to run infra which just sits idle when there are no tasks. Would be great to provision pods on demand. Is it possible today?

no, but that will be a great enhancement and Pinot has the primitives to achieve that

feel free to start an issue around this, I will add my thoughts and provide some pointers

Does Pinot do early stop when enough results have been already collected? We have queries of form "SELECT * FROM table WHERE userID='H' AND sourceEventTimestamp>=t1 AND sourceEventTimestamp<=t2 ORDER BY sourceEventTimestamp DESC LIMIT 500". The table has been sorted by sourceEventTimestamp and userID has inverted index. I notice that the selectivity of the query is low (meaning many rows passing the condition). So the first 500 results should be collected relatively quick. But the exec times are long i.e., > 10s.