Hello everybody, really appreciate your help/point...
# general
j
Hello everybody, really appreciate your help/pointer on this one. I have a performance question, regarding getting a deep-storage query result back from a durable-storage S3 location. Currently using the “druid/v2/sql/statements/{queryId}/results/page=0&&resultFormat=csv” endpoint, it takes approximately 30 minutes to download 1G of query result, using a curl HTTPS call running inside the same AWS region (as the S3 bucket) My questions are; 1) whether this download speed is a typical performance for getting deep-storage query results 2) are there any configuration tune-ups on the Druid settings to improve this deep-storage-query flow. Thanks.
c
hey bro, I want to configure the deep storage druid as local disk, and use druable storage as minio (s3), is that okay? My druid cluster only in local server, I would like to use result pagging for deep-storage queries.
u
hey Jamie. Even in our experiments with Druid Deep storage execution and then download of results took >10mins. This varied with load on the system, as this was the performance noted during high querying hours.
Do verify whether the query execution is taking that long vs download taking that long. We saw execution itself take up to 10 mins. (Not sure what the data size for these queries were)
j
Thanks Utkarsh. The download does take longer than query itself, was just wondering if there is any feature release in the pipeline to speed up the getting the whole result from deep-storage queries.