https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
u
Any pointers where i can read or understand it a bit more. We are trying to improve the perf of our queries by re-ordering the columns in such a way so that similar data co-locate.
a
For string dimensions, druid stores bitmaps for unique values that improve the filtering performance dramatically. The string dimension is dictionary encoded and searching a value within that dictionary does use binary search.
long dimensions on the other hand do not have bitmaps so there is definitely some slowness there
One trick that I have seen people use, is to store long columns as a string dimension instead. and thus reap the fruits of bitmap indexing when filtering on the said dimension.
there is work in progress that will obviate the need for this shenanigan in future druid release.
u
how does the filtering works (equal + range) on long column then ? Binary search ? or it is just wild guess . We are doing range based queries on the long column , i am guessing if we change it to string, it will have impact on perf ?
a
it will evaluate each row if that matches the filter you have passed.
yes. if you change it to string, you should see perf impact
u
sorry to trouble you again. i have one more question. so my segment has this dimension spec -> 
__time, dim1, secondaryTime
. case 1: Segment has only one value for dim1, hence secondaryTime would be in increasing order and data would be sorted completely in segment. case 2: Segment has around 10-20 values for dim1, Even though secondaryTime would be increasing but the overall data would not be sorted, as data is sorted by dim1 first. Amount of data/rows are same in both segment. query -> select count(*) from datasource WHERE 
secondaryTime > 10000
I see that queries returned faster result for case-1 segment. Does this mean if data is sorted, filter works faster on long columns ?
a
how much faster it is?
u
approx two times.
one guy took around 450ms, and other one took 750ms.
a
hmm. don't know for sure. when the secondaryTime is sorted, there is more accurate branch prediction. Though I don't know if that's driving this difference or there is something else.
wrong thread 🙂
u
hmm, got it okay. meanwhile chatgpt is on different level with it's knowledge about druid. it says druid using roaring bitmap for numerical columns 🙂
🙂 1
c
heh, it lies, unless it is talking about numeric columns created with the nested column indexer 😅
what is the size difference between the two cases?
u
dataset size is same.
c
the segment sizes? typically when data is sorted it also tends to compress better resulting in smaller segments
with long columns created by traditional means today, filtering happens doing a full scan
u
segment size is around 700MB.
c
numeric columns created by the nested column indexer have value indexes, so a range filter does a binary search of the sorted value dictionary to find the start and ends of the range and then unions all of the bitmaps for those values together
string columns have similar behavior, but using a lexicographic ordering, so i think numeric values stored in strings doing numeric ranges fall back to a predicate index that instead matches every value in the dictionary
https://github.com/apache/druid/pull/12830 has some details about the numeric range stuff
using indexes isn’t always faster when the range results in a ton of bitmap operations, the threshold i observed was around 200k values in the range before it got slower than using the matcher, so will probably update in the future to put some smarts around that
that write-up is a bit string centric, but much of the same applies to the indexes used by json columns
u
thank you so much. This is what i was looking for to understand how filtering works for all sorts.
NumericRangeIndex -> is this something going to implemented for normal numerical columns as well ?
c
i want to add range-only indexes to numeric columns as an alternative to having an index per value to better serve higher cardinality numeric columns as well, but I haven’t got to it yet
the idea being we have an index for all the values in some range and then use a value matcher to filter down the values that fall within the buckets
yeah, in one way or another
ive been making some changes to the nested column indexer to make it into a ‘auto’ column indexer
since it needs to discover type/schema information within nested columns, it can do the same thing when the data isn’t nested to power schema auto discovery based ingestion, which means it will be writing out a long column when it processes only longs or string for only strings, etc
which means it will be able to be used for normal ingestion as well, just substitute ‘long’ for ‘json’ (or maybe ‘auto’ or ‘built-in’ or something) once we are done making the improvements
undecided if i am going to modify the existing numeric dimension indexers, or just add type coercion to the json indexer, so you can enforce that all the values are a LONG or whatever other type
u
sorry, but i did not get it completely, not much familiar with the nested column indexer yet. I will go though the links you shared to understand it a bit more.
c
sorry i guess we got off track a bit, my guess would have been that because the data was better ordered, the smaller column size was able to be read faster, but if there is no difference then it is unlikely to be that
u
hmm.
c
if you have the unzipped segment files, there is a 
meta.smoosh
inside that has the byte sizes (in the form of offsets) for the column stuff
u
another question -> Another experiment data size is same. case 1: data is distributed in 240 segments. case 2: data is distributed in 20 segments `query -> select count(*) from datasource WHERE `secondaryTime > 10000`` when i fire the same query on long column, case 2 is faster. (this is also 2 times faster) does it mean lower number of segment will return result faster, even though the data set size is same.
r
the number of segments will impact the performance as well, if you run autocompaction if they are in the past and the data is not changing anymore this will help reduce the number of segments down until they are large enough to the overhead of the number of segments be mininal
u
hmm, indeed i find in my experiments that when i ran queries on same data re-indexed in less number of segments, queries are running faster.
Open in Slack
PreviousNext
Powered by