I ingested close to 200Gb of data without providing any indexing and im assuming by default pinot used dictionary encoding to all the columns. My questions are 1. can i change the index of a couple of columns to sorted index by updating the table config and reloading the segments? 2. If yes, will it take time to reload all the segments- ingestion took around 15hrs. Does this cause any downtime to the cluster? 3. Is there any documentation on which indexing should be applied to different types of columns- like based on cardinality or filtering conditions?

Hi Rohit, Yes you can add and remove any columns from any index. Just change the table config and click reload. The operation should be pretty fast, ideally finishing within less than 5 minutes. No downtime will be caused because of it. The best practice is to keep replication > 1.

@Kartik Khare thanks. Replication is greater than 1. Also is there any documentation on how to choose the index based on cardinality/ query predicate/ data type etc?

generally the best way is to created inverted index on columns which will be used in WHERE clause with = sign then create range index on columns which will be used with < or > sign other than these two the rest should be added based on the query performance

Okay. Understood. Also is it prudent/ possible to disable indexing on high cardinality columns that arent used in any predicates (data already ingested)

@Rohit Anilkumar This gives you a high level understanding of which index can be used for which case --> https://dev.startree.ai/docs/startree-enterprise-edition/startree-dataset-manager/recipes/indexes

@Jayesh Asrani thanks. Will take a look. Can you also answer the previous question I have asked. @Kartik Khare

Yes you can remove the indexes if not being used in any queries. Curious the idea is to reduce disk footprint by removing?

Im just trying to understand all the cases. So i have some 200gb data ingested with no indexing and the query took 6sec. Im trying to reduce the runtime. So maybe I will alter the indexing for the columns that are used in the query filtering and see if that helps. And im assuming if i dont do indexing on the high cardinality, less used columns, it might reduce the disk usage and even if i use it, there wont be any reconversion of the encoded columns to actual value. Will try this out anyway

Got it, 👍