Hi! I have question about updating indexes without re-ingestion data in segment-store. I mean in documentation there is such information:

The updated index config won't be picked up unless we invoke the reload API.

I have added new index(

inverted

) to table config and call reload API(with

forceDownload

flag) but as I see index is not created(I checked in

reload status

and even in

metadata.properties

file of particular segment). But after I re-ingest data(with spark job) to segment-store then index is created. All these operations are done for

offline

table So the question is: is it necessary to comply some additional conditions to make reload API working? Maybe, I missed something

@User The reload api should suffice. Depending on the size of your data, it might take a bit of time to generate all the indicies. What did the reload status say?

Solved in PM. Just for history: the main problem was that I have raw index(no dictionary) on column that I want to add inverted index. But reload doesn’t work for adding dictionary-encoded forward index because it need to rebuild segment fully Thanks @User!

Thanks @User for the summary. @User for help with doc improvement.

@User what's the difference between 'reload API' and 'rebuild segment' - is there a swagger end point for rebuilding the segment? If we click 'Reload all segments' in the Data Explorer, what does that do?

They are the same. Essentially, once we change indexing configs to add/remove indexes, we call reload api (or rebuild) for it to take effect.

But reload doesn’t work for adding dictionary-encoded forward index because it need to rebuild segment fully

So for this scenario you have to build the segment again manually?

Yes, here I mean regenerate segment(in my case call spark ingestion job again)

Have updated the Indexing FAQ to explain the above - https://docs.pinot.apache.org/basics/getting-started/frequent-questions/ingestion-faq#indexing Also added a link to the FAQ from the indexing page - https://docs.pinot.apache.org/basics/indexing @User let me know if I missed anything!

Looks good, thanks! One thing is that in section https://docs.pinot.apache.org/basics/getting-started/frequent-questions/ingestion-faq#can-i-retrospectively-add-an-index-to-any-segment links sorted index column and the dictionary encoding of the default forward index are going to 404 page(at least for me)

oooh thanks, I'll have a look what I've done wrong with those links!