I am trying to use pinot startree on queries that have a range filter on my time column. they look like:

select max(time_column), count(*) from table where star_tree_indexed_col = "some value" and time_column >= 1000 and time_column <= 2000

time_column is not a startree dimension, just the timecolumn for segment pruning. The pinot segment min/max for timecolumn should show the selected segment trivially is a subset of the time filter. I found that this correctly enables startree only if time_column has a dictionnary index. It performs full scan otherwise. I tried adding a range index and it would do a range filter scan. What is the reason for this? Would a dictionary index on a timestamp (millis) column be a big impact on storage?

The current implementation does not support non dictionary based predicate evaluators inside of StarTree. https://github.com/apache/pinot/blob/b396056caf9cdd5c2a32b8790a46641908d73247/pino[…]src/main/java/org/apache/pinot/core/startree/StarTreeUtils.java

I think it's just the way that table time column has been implemented. we need that to be granular to make some precise queries when needed (usually fetching days in different timezones). otherwise we write segments containing UTC days slices of data... do you mean it would have been better to have timestamps truncated to a UTC day as a time column?

Too much is relative, but I won't rule it out if you need to have it.

good point. seconds may be fine

Check swagger, there should be an api. If not, then

index_map

file inside the segment has it.

I wasn't able to find a swagger endpoint for it or that index_map file in the segment deep stores... I might have been looking at the wrong places 🤷 (pinot 1.4.0)

Index map file is inside a segment. Deepstore has tar.gz. Easier would be to check server directory. I’ll find swagger api when I am in front of my Mac