Hello, We have a production setup where data is sent to a Kafka topic, which is ingested in Druid. We are interested in doing some more exhaustive data analysis, and wanted to use other tools like Spark, Pandas or some machine learning tools to find patterns in our raw data over the year or make predictions about future data. Imagine the simple case where I want to analyze the data for the last year, so I need to download the columns for that data over that range of dates, which will be billions of rows. I wonder what is the usual way of doing that when data is stored in Druid. I can think of some ways to do it, but none of them probably is something good enough. • option 1: have a script that pulls all data from druid pulling the data in chunks

SELECT from my_datasource WHERE __time >= :start AND __time < :end LIMIT 50000

. But

__time

is not unique, so that is a little bit annoying to do, and pretty slow. • option 2: data is stored in S3, so maybe we could pull the data from S3 and try to read the format stored there (not even sure if it is possible, but it seems problematic at least to support this over the year if Druid changes the storage format) • option 3: have another Kafka consumer group and read the same data from the Kakfa topic and store it in a different place where we can read it easily. This would only work for future data, but not for data already stored in Druid. • option 4: queries from deep storage (https://druid.apache.org/docs/latest/querying/query-deep-storage/) but I don't know if they are a good idea for pulling so massive amount of data • option 5: Integrate druid with other tools like Spark, as explained in https://druid.apache.org/docs/latest/comparisons/druid-vs-spark/ but to be honest, I don't understand this approach much, and I find it it may be over-engineering for just pulling a copy of all data in a timeframe Can someone tell what is the usual way of doing this? Thanks.