Objective: Generate star tree on demand so high level design • Create a task generator (StarTreeIndexTaskGenerator) in controller • OnEveryTableConfig and/or periodically it gets the metadata of segment from server and compares it with tableConfig • If there is a difference, it downloads the segment, invokes SegmentGenerationJob and pushes the segment back • Trigger segment reload

why is

trigger segment reload

needed at the end?

Question, so we don’t allow server to create startree index on the fly, it’s all coming from controller?

also, why doesn't star tree work like inverted indexes, wherein just updating table config and reloading works?

probably not needed if the CRC changes

i thought its needed bcos the segment name remains the same and servers dont download the new segments

It can but it is resource intensive and might impact query performance

@User has left the channel

Huh.. Seems like the message history in this channel is lost. Is this archived somewhere by any chance? I’m interested in knowing more about the minion usage and its stability.

any design docs or wiki is fine too

@User ^^ any design doc you can share about minion?

@User Are you planning to implement some customized task for minion?

AFAIK it is only used in LinkedIn right now to help purge records for GDPR compliance. It is quite stable and is powering the production use cases.

FYI, that part of the code is not open-sourced because it integrates with the LinkedIn internal services.

You may refer to this test

SimpleMinionClusterIntegrationTest

to understand how to add new task for minion.

Yes @User The use case is almost like I want to purge data based on the value of a column.

Oh, actually we have the executor open-sourced:

PurgeTaskExecutor

Where you can register your own

RecordPurgerFactory

Unfortunately we don't have doc on this part yet. We can try to make it work, then use it as an example to add the doc, or make a post

okay. thanks. I just got this idea and wanted to explore. Seems like it’s possible. We may or may not implement immediately but will definitely ping here if we’re proceeding with impl. appreciate the help

@User has left the channel