This message was deleted.

We have this experiment right now, actually. So theoretically you can, but in cases with "random" string values you might have huge cardinality. Basically, you will need a bigger cluster to handle the data than in Presto. Maybe with some data retention rules it would be mitigated. Makes sense?

Yup! We are on the same wavelength. I want to solve data discoverability and experimentation while running only 1 type of cluster for ease of deployment. I am thinking of limiting retention to only 2 weeks or 1 month to manage costs.

Or, use the presto/trino druid connector?

what you mean by data stored in Presto? @Michael Kutz isn't it just a proxy/engine? do you mean druid compared to data stored in object-storage?

So, within our group, data is first ingested into a data lake powered by Iceberg. We then installed Trino on top of that Iceberg catalog for data discovery and experimentation. Now, over time, we find that Trino is really quite slow and our users are getting frustrated. This is why I am wondering if Druid can simply fill the gap. Also, Because of having both Trino and Druid, our users have to write the same thing twice.

yeah, we have pretty much the same setup. I'm far from claiming that I understand how Druid works, but I think it boils down do costs. There are use-cases that Druid will have trouble, weird functions that have lots of data on the broker or full-text search, and there Trino will be also slow but you expect it to be slow lol. Maybe you can run Trino over Druid and have a "quick access" table that is druid and one slow access table that is iceberg? This way you can use one sql syntax for everything?

my 2¢: i think what you're asking is really: can druid be used as a primary, general-purpose analytical database across an organization? i've seen some people do this with wild success! and i've seen some people attempt this and run into issues; in this second camp, the issues people tend to run into are: 1) SQL support — druid does not support everything in SQL yet. some gaps people cite include joins that require shuffles [druid joins, today, broadcast all but the largest relation], and windowed aggregations 2) cost for infrequently-queried data — druid requires a copy of data, in druid segment format, to be stored on data servers before it can be queried. this works great for frequently-queried datasets, since it's a very high-perf approach. but for infrequently-queried datasets, it can feel wasteful of storage. generally, that first camp of people -- the ones with wild success -- are people that don't have these two items come up in their situation for the second camp, @ imply we're actually working actively on these items; for some details on the vision check out: https://imply.io/blog/a-new-shape-for-apache-druid/ the first fruit of this work that you'll see is SQL-based ingest, coming in the next Druid release. you'll see more stuff coming in future releases too, so stay tuned 🙂

Or, use the presto/trino druid connector?

on this idea specifically ☝️ i don't have much experience w/ the connector, but i worry about the overhead it adds to the queries that get routed to Druid, especially in high-concurrency scenarios (Druid is designed to be able to handle hundreds-to-thousands of QPS; from what i've heard, i don't think presto/trino does well with this kind of workload). although, if your expected druid workload is lower concurrency, it should work OK. and it's certainly part of the allure of presto/trino: one system for querying data, using one dialect, wherever it lives.

@Gian Merlino - absolutely not intended as a snarky remark. I am genuinely interested to know how folks have achieved 1000s of qps in Druid. We always ran into either historicals running hot on cpus or there not being enough broker threads to keep up with the qps. Note that this was with a fairly heavily loaded cluster with no tiering and using top of the line hardware. I am assuming this involved almost all of the data in page cache which reduced segment scan time and other associated metrics significantly.

for the bit about 1000+ QPS: it's generally a situation where most queries hit a slice of the dataset, and things are set up such that pruning can be done at the broker level. meaning: queries naturally fall along datasource lines (and the data are partitioned by datasource; and there's lots of datasources) or they naturally fall along dimension-value lines (think: filter by userid = X and the data are partitioned using ranges of userid). this pruning at the broker level means that most queries only hit a handful of historicals each, which enables very high concurrency as each historical is only seeing a portion of the overall queries

So the sauce is metadata caching and right kind of partitioning 😁. Sounds familiar, but obviously needs a bit of thought when designing the schema.

for the question about getting good query times without rollup: it's kind of a common misconception that Druid is fast because of rollup. rollup is definitely helpful, but even without rollup Druid is quite fast compared to Presto/Trino due to the efficiency of the storage format and query engine. like, when measured on simple metric "how many rows can we process per CPU per second", Druid posts excellent numbers. where it's less competitive is queries that involve multiple stages of processing, as that blog I linked discusses, & which is a major focus of our effort over the next year

Just read the blog. Exciting times in Druid! Happy to see all the work especially the downloading from s3 for querying.

yeah, that's true & that's why we have been recommending

range

partitioning recently (now that we support multi-dim range partitioning)