https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
r
for what I read so far, it's not worth long search sequences unless your prefix patterns are also very large (eg: 128 chars)
r
cool, I'll check these out. Thanks!
r
At first I thought it was some parameter like window memory for LZ77/LZ78 but it's actually more similar to a data struct (hence encoding in it's name 🤣) rather than compression
c
yeah, the 
bucketSize
parameter is the number of entries per bucket. many filtering operations require looking up the values in the dictionary, which for front coding works by doing a binary search of the first value in each bucket, then a linear scan to find the specific value within the bucket. The performance growth is a bit complicated, since the larger buckets make the binary search phase a bit smaller, but the larger buckets to decode does add up a bit
i went with 4 as the default since its performance was closest to the existing implementation while still showing pretty nice size benefit in a lot of cases, but 8 and 16 are pretty effective as well without too much performance impact
i didn’t test much larger bucket sizes
eventually i plan to allow this to be configurable per column rather than at the segment level, and i’d also like to add an ‘auto’ mode that choses the ‘best’ bucket size for a column when building segments
since it isn’t very beneficial to use this for very low cardinality columns, and at the opposite end, higher cardinality columns would probably benefit from larger bucket sizes
r
ah cool. Thanks for the detailed explanation!
4 Views
Open in Slack
PreviousNext
Powered by