Hi all! I've been reading about Druid and I need to evaluate how it fits our usecase. We need to ingest a lot of data. I really mean a lot. About 28bn rows per day. I see that druid does a lot in terms of horizontal scaling. There is lot of documentation which focused on data ingestion. However, I have way harder time finding information about ways to pull data out of Druid. We have our normal analytical usecases where aggregations are performed. Rollups are a nice feature which will help us reduce the amount of data over long periods of time. One of our other requirements revolve around data extracts. We need to be able to retrieve the raw data out. Normal usecase is that basically everything that goes in, needs to come out as well. Normal functioning requires daily extract of the 24h period. In very rare occasions it might be a multi day. The extract doesn't need to be a single query, but it also ideally wouldn't be millions of queries. We also have tight timelines where the full day's extract should be done in about 4h. I'm having trouble understanding the capabilities on this side. Some the things that I'm interested in are: 1. Are queries themselves distributed across multiple machines, similar to what Spark/Presto/Trino do? 2. Can result of queries be pushed into storage like S3 directly by Druid and avoid client paginating? Something similar to what AWS Athena does. 3. Are there any limits which prevent/complicate this task? If yes, how? 4. Anything else that I should know. Is this a bad fit for Druid in general and should keep this on the datalake and look to Druid as additional capability. Side question: 5. The raw data needs to be stored for up to 1 year! Any thought here are appreciated!

Are queries themselves distributed across multiple machines, similar to what Spark/Presto/Trino do?

yes data is colocated with the compute so you would get faster turn around times. We have 2 engines : 1. Classic : Scatter gather engine for queries which need reponses in sub second. 2. MSQ: Shuffle based engine for which queries need answers in minutes. a. We also have a third engine profile based on MSQ called DART which is all the good stuff from both engines.

Imply DevRel here - if ya wanna talk it through @HEPBO3AH then happy to catch 30 mins with you: https://calendly.com/druidcommunity/getstarted

Thanks @Karan Kumar. Regarding the Export to parquet (ideally because of the data volumes) I cannot find it in the documentation. Do you mind flicking a link to me? The link I managed to find is the Apache Parquet Extension, which is related to import not export. In general, it would be great if you got any documentation I can look at. I apologize for my really bad googling skills, but I just cannot find anything official on these

We donot have the apache parquet export yet. Its a todo but we never really got around to implement it. Hoping you could pick it up 🙂

Thanks! We would really like ability to at least have the data compressed due to it's size. Normally we get 90%+ size reduction with compression

You mean data compression on export ?

Yes

that is not supported iirc