https://linen.dev logo
Join Slack
Powered by
Hi, if we want to run large reports on Druid which...
# general
k
Hi, if we want to run large reports on Druid which can contain like 2M rows to 40 M rows . what are some recommendations we started testing with 2 M rows but we are seeing this error on broker 
org.eclipse.jetty.io.EofException
. Query finished on historicals so thinking broker got this error while transferring data back to client .
u
I think you should look for MSQ based queries. these could long running jobs.
Have not used it for queries but i saw some wherethis fits pretty well for long running/ reporting kinds of use case.
k
but msq is used only if we use deepstorage querying . but looks like it will work only if all the data for the time interval is present in deep storage it won't work if some part of data is present on historicals
u
no. it works just like any other queries.
k
u
i think he meant that it works by downloading the segments from deep storage. (It does not consider any other segment which is not pushed to deep storage though which means can not query real time segments) @John Kowtko Is my understanding correct here ?
j
Hi Udit, Krishna, Other than real-time, ALL segments are in deep storage, whether or not they are loaded onto Historicals. MSQ originally would only query deep storage and could not access the real-time segments (from streaming ingestion tasks), but that was changed a while back, you can now use the 
includeSegmentSource
context parameter to instruct MSQ to also query the real-time segments. Doc ref: https://druid.apache.org/docs/latest/multi-stage-query/reference#context-parameters
k
So if understand correctly if I have a query with last 3 months interval and datasource has a retention of 1 year and we store only 2 months in historical and rest of the 10 months in deep storage will this deep segment msq works ?
j
Yes, MSQ will be able to access the full year's worth of data. Web console will only row counts and segment size for what's loaded on the historical, though ... but the availability column should show up that there is some loaded and some not loaded data.
k
Thanks @John Kowtko
@Udit Sharma / @John Kowtko can you please provide few examples how to submit query using msql to query from deep storage and retrive the data back . I have used this but it is not working for me
Copy code
curl --location --request GET '<http://localhost:4080/druid/v2/sql/statements/query-3850b19b-8913-4a27-9eed-526523b47098>' --header 'Content-Type: application/json'
{"error":"druidException","errorCode":"notFound","persona":"USER","category":"NOT_FOUND","errorMessage":"Query [query-3850b19b-8913-4a27-9eed-526523b47098] was not found. The query details are no longer present or might not be of the type [query_controller]. Verify that the id is correct.","context":{}}
j
Here ... you can pull the example from a small bash test script I wrote for the three query APIs ...
apiPerfTest.sh
k
Thanks John , will go through this
3 Views
Open in Slack
PreviousNext
Powered by