https://pinot.apache.org/ logo
#troubleshooting
Title
# troubleshooting
p

Prashant Pandey

06/08/2022, 5:04 AM
Hi team, if I add an index to my table’s indexing config, what happens to the old segments? Will the new index be created when the segment is loaded during query? Do I need to reload all the segments?
n

Neha Pawar

06/08/2022, 5:35 AM
index will not be created on old segments during query. You need to reload for it to be effectve on older segments
🙏 1
p

Prashant Pandey

06/08/2022, 11:06 AM
Thanks
@Neha Pawar Just for my understanding, reloading takes the latest table config and applies it to all the segments? Does it all the segment into memory as well? Can I run into OOO if I have too many segments for my table?
n

Neha Pawar

06/08/2022, 2:39 PM
Not exactly latest table config. It can do some things only like new column, derived column, new index, remove index. It cannot change encoding, drop column.
It doesn't bring the whole segment into memory. It only builds the new index, and attaches it to the existing segment's metadata
@Mark Needham do you have a reload recipe that explains what it can and cannot do and how?
m

Mark Needham

06/08/2022, 2:44 PM
nope, but that sounds like a good one
what does 'derived column' mean btw?
p

Prashant Pandey

06/08/2022, 2:47 PM
@Neha Pawar Thanks this helps!
n

Neha Pawar

06/08/2022, 2:48 PM
Same as transform function, but done on a live table, where old segments get their new column value backfilled from existing Vikings
m

Mark Needham

06/08/2022, 2:49 PM
so to add a derived column: • Add transform function with new column • Reload segment ?
n

Neha Pawar

06/08/2022, 5:47 PM
yup!
h

harnoor

07/14/2022, 9:58 AM
What if we add a sorted index on a column? In that case segment needs to be sorted and will it be loaded in memory?