This message was deleted.

fwiw, I'm a maintainer of Vitess and a top-100 ish consumer of BigQuery, so I have context into systems similar to Druid, and know why they aren't a good fit. I know there are often issues when you're using a system differently than the initial design, even if it seems to be a fit on the surface

Hi Derek, It's not Druid's sweet spot, but it might be worth a try. HTML strings would likely need to be stored in STRING fields in Druid. The STRING field in Druid by default is dictionary encoded ... which works great for low to reasonably high cardinality with limited length values, but in the case of HTML pages the dictionary would be composed of extremely long and extremely unique strings. (doc ref: https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion#indexspec) The dictionary can be front-coded, the segment files are zipped when they are stored in deep storage ... so you would get some savings there. And you would be able to store large numbers of records together in the same segment files. You can handle the vast amounts of historical data by ingesting them and storing on deep storage only ... the queries against them are slower, but you don't have to allocate data nodes to pre-cache the data. (doc ref: https://druid.apache.org/docs/latest/querying/query-deep-storage/)

@John Kowtko thanks for the detailed response - it's very helpful to know that it's not going to immediately cause issues, even if it's outside the sweet spot

@Derek Perkins if you try it, let us know how it goes! btw Vitess is cool - I first heard about it long ago when I worked for MySQL, and we visited youtube and got a presentation about it. Very nice 🙂

thanks @Ben Krug. I think we're leaning towards using Apache Iceberg as the storage format, which will let us continue to use BigQuery, but also investigate other engines that meet our realtime needs better. I saw that Druid now supports it as of v27 too, which is awesome. https://github.com/apache/druid/pull/14329

yeah, iceberg, and querying directly from deep storage, etc, lots of new good stuff coming out. Is Iceberg in 27 or 28? PR seems to say 28?

it was listed in the v27 release notes