Sorry to be a never-ending fount of questions, fol...
# generalw
Will Briggs
01/05/2021, 4:46 AMSorry to be a never-ending fount of questions, folks… is it expected / necessary to create a rangeIndex on dateTime fields, or are those automatically indexed efficiently? Likewise, should I add dateTime fields to the noDictionaryColumns list?
m
Mayank
01/05/2021, 4:47 AM
What's your time granularity?
Mayank
01/05/2021, 4:48 AM
Typically, we don't need to set explicit indexing for dateTime fields, as we can still prune segments based on metadata.
w
Will Briggs
01/05/2021, 4:49 AMMy base timestamp is epoch milliseconds, I am playing around with deriving an hourly grain field for pre-aggregating in a star-tree index, but based on my current prototype, that seems like it might be premature optimization
m
Mayank
01/05/2021, 4:50 AM
A general recommendation is to sort on primary key (or a dimension that appears in most queries), and minimal number of inv indexing to have a reasonable selectivity across your query set.
w
Will Briggs
01/05/2021, 4:51 AM👍
Will Briggs
01/05/2021, 4:52 AMThank you. I am also looking to partition the incoming data based on a dimension that is almost always used selectively in the WHERE clause, and use broker-side partition pruning to minimize scanning unnecessary segments - I’m not sure if that will actually help, or if forcing data locality like that will bottleneck things.
Will Briggs
01/05/2021, 4:53 AMI am using that same dimension as my sort key, but right now, it’s not particularly useful to sort on it, because it shows up in all segments
m
Mayank
01/05/2021, 4:53 AM
Oh yeah, partitioning is definitely a good idea.
Mayank
01/05/2021, 4:53 AM
In our usecases, we typically have the partitioning as well as sorting on the same dimension
w
Will Briggs
01/05/2021, 4:54 AMOk, that makes me feel better, as that was my plan.
m
Mayank
01/05/2021, 4:54 AM
Good plan, I'd say.
w
Will Briggs
01/05/2021, 4:55 AMAnother stupid question - should I create an inverted index on the sort column, or is that unnecessary?
m
Mayank
01/05/2021, 4:55 AM
That is unnecessary, it won't be used. In fact, the segment generation might just ignore and not create.
w
Will Briggs
01/05/2021, 4:57 AMPerfect, thank you. It seemed like it would be unnecessary, but I’ve seen stranger things, and the docs, while great for an incubating project, were a little unclear - I would love to volunteer to keep notes while I’m doing this, and maybe propose some updates to the docs if that would be helpful.
👍 2
m
Mayank
01/05/2021, 4:57 AM
That would be really awesome, would really appreciate your help in improving our docs.
Kishore G
01/05/2021, 5:40 AM
it talks about all the indexing techniques and when to use what.
w
Will Briggs
01/05/2021, 5:41 AMThat’s awesome, thank you