Andres
10/24/2023, 3:30 PM"select count(distinct id) where country = 'US' and ((arrayContainsInt(categories, 2) and arrayContainsInt(categories, 8)) or arrayContainsInt(categories, 25)", these queries can get complex with multiple and/or with the categories array.
I would approach this with set intersection/unions, by aggregating sets with devices ids for each country/category combination. Thetasketch support is great in Pinot but won't work in my case because I will be intersecting large sketches (around 1b unique devices) with very small ones (around 50k unique devices) and the error rate is too high. My second option would be to do it with roaring bitmaps, but AFAIK pinot doesn't have support for them? The star tree index has a bitmap aggregation but I'm not sure how to use it, is it possible to do union/intersections with that? Is there any other way to solve this? My requirement is for queries to resolve in less than 500ms in an ideal case, I can tolerate up to 1 second for heavier queries.