https://linen.dev logo
Join Slack
Powered by
Hello Druid Experts, Has anyone performed benchmar...
# general
v
Hello Druid Experts, Has anyone performed benchmarks specifically for the “Virtual Storage” feature in Druid version 35? I’m interested in understanding the query latency metrics when Virtual Storage is enabled. Also, since this feature is marked as experimental in version 35, is it expected to be production-ready in the next release?
c
i haven’t done a lot of benchmarking yet as there is still quite a lot of work in progress; the current experimental release version has some limitations such as it only works with the ‘classic’ druid engines (no support for the newer ‘dart’ engine) and requires that all of the segments that are taking part in a query be downloaded prior to performing any processing
we are working on dart engine support with a much nicer integration that interleaves downloads with processing work, which should be much more efficient
i will also be raising a PR probably this week that introduces a new segment format that is built for being able to support partial downloads of segments when coupled with 
druid.storage.zip=false
(stores segments as uncompressed files instead of a zip, which s3 supports, but i don’t believe other deep storages do yet..)
query engine support for partial downloading of only the segment parts that are needed to perform the query will come at some point later
running with 
druid.storage.zip=false
also helps quite a lot with on demand download performance since we don’t have to have the overhead of decompressing the segment to map it, so i would recommend doing this if using s3 and you want to experiment with virtual storage now
i believe the segment format and maybe dart support will make it into druid 36, partial downloads will be later since there is some work to do there
Open in Slack
PreviousNext
Powered by