you can apply it on old tables then reload all the segments

will it be stored in ram or ssd

it will be off-heap

I mean i onl;y need to apply the indexing on the string columns or anything else also needed

apply index, then it will improve the text search query

but you pay extra disk space for index

that will also improve latency drastically i guess

since we are also doing filter with substring on a string column

so lucene fst on that string column where we are doing substring while filter will help?

apply index, then it will improve the text search query => will regexp query or substring query performance wont improve if i use lucene fst and only will improve if i use text_match clause ?

cc: @Mayank @Jackie @Subbu Subramaniam @Kishore G @Neha Pawar @Ken Krugler @Daniel Lavoie

cc: @Mohamed Kashifuddin @Shailesh Jha @Mohamed Hussain

substring won’t get benefit or regex

if time column is primary time column in schema and stored as long data type in milliseconds epoch with granularity as seconds .. is it necessary to use range indexing on time column on my tables if all queries have filter(where clause) with respect to time say last 5 min or last 1 hour or last 30 days @Xiang Fu.. I mean by default segments do have start time and end time if time is primary time column I GUESS .. and segment pruning will happen by default or range indexing on time column is needed?

do you have sort index

you can sort time

but the time coming on kafka topic may not be sorted .. I mean some unordered /unsorted time may come while ingestion

so is sorted index recommeded in such a scenario?

for time column

but you can only pick one sorted index

then range indexing wont be needed on time column .. right

also sorted indexing will internally also be doing inverted indexing

also one last doubt .. inverted indexing performance will be same for string type columns and long/int columns or int/long columns are going to give better performance even after indexing applied .. my use case is I have some tenantId or other ID columns as string but they are digits only .. so is it necessary to keep them as long type in schema and then only apply inverted indexing or I can even use string data type with inverted indexing and it wont have much impact on query latency while filtering or group by ..

HI @Subbu Subramaniam @Xiang Fu @Jackie @Mayank: will inverted indexing also help for IN clause query or only for equals?

also is it possible to remove indexing from any partiular column in the table and reload all segments like suppose in future i dont want lucene fst indexing on one string type column say col1.. so can i edit the table and remove that indexing and reload all segments and it wont cause much trouble?

also if the table schema is changed by adding a column .. then what impact it will have on existing indexes and can i add indexing on the newly added column @Xiang Fu

Right now, delete indexing won't delete physical index on disk

No trouble, just more disk usage

I mean first step will be editing schema and adding a column > reload all segments > edit table > add inverted indexing on newly added column > again reload all segments => will it cause any issue?

your steps will work