Hi all! I'm doing a POC with Pinot and I need some...
# general
a
Hi all! I'm doing a POC with Pinot and I need some help to approach my usecase. I have an event stream with an event every time there's any activity in a device, the event has a device id, a country and an array of integer categories (between 1k and 2k integers). I get around 17k events/s and 10b unique device ids. My queries are something like (just an example, not an actual pinot query)
"select count(distinct id) where country = 'US' and ((arrayContainsInt(categories, 2) and arrayContainsInt(categories, 8)) or arrayContainsInt(categories, 25)"
, these queries can get complex with multiple and/or with the categories array. I would approach this with set intersection/unions, by aggregating sets with devices ids for each country/category combination. Thetasketch support is great in Pinot but won't work in my case because I will be intersecting large sketches (around 1b unique devices) with very small ones (around 50k unique devices) and the error rate is too high. My second option would be to do it with roaring bitmaps, but AFAIK pinot doesn't have support for them? The star tree index has a bitmap aggregation but I'm not sure how to use it, is it possible to do union/intersections with that? Is there any other way to solve this? My requirement is for queries to resolve in less than 500ms in an ideal case, I can tolerate up to 1 second for heavier queries.