Apache Pinot

<@UDT7GFEG6> <@U02DXKP9B6C>
Regarding this comment : <https://github.com/RoaringBitmap/RoaringBitmap/issues/608#issuecomment-1364722016>

I have added the getContainerSize() method in Roaring bitmap : <https://github.com/RoaringBitmap/RoaringBitmap/pull/742>

Can we have a thread here , what all can we improve in pinot with this  ?

```There's a place in AndDocIdSet where the bitmaps are sorted by cardinality (smallest to largest) and that sorting would be a lot faster if sorted by container count as a proxy for cardinality.
I'm not sure how much impact this would have though, and it's not a perfect proxy for cardinality, you'd have to test it```
-Richard

<https://github.com/apache/pinot/blob/9748dd06a45e4143d1d0f7353ff8ffe2160d62fd/pinot-core/src/main/java/org/apache/pinot/core/operator/docidsets/AndDocIdSet.java#L110|https://github.com/apache/pinot/blob/9748dd06a45e4143d1d0f7353ff8ffe2160d62fd/pino[…]/java/org/apache/pinot/core/operator/docidsets/AndDocIdSet.java>

<@UDT7GFEG6> Let me know if you have any inputs here

After some more findings realized that While container size could offer a faster sorting process, the overall impact on intersection performance could be negative, especially with datasets that have varying densities of bitmaps. Cardinality remains a more reliable and performance-friendly choice for sorting in this context.

Calculating cardinality itself can be expensive, so one low hanging optimization would be cache the cardinality to avoid it being calculated multiple times

Okay will explore that , do you think it will be worth to give it a try with size ? 
Richard still hopes that it might work out and asked to me to try 

Sure. Seems like the overall goal is to perform an AND to all the bitmaps. Richard is the expert on that so I'd follow his suggestions

hey, just saw this thread. I don't use the other account any more. As I mentioned to Anand, I'm ambivalent about the approach. OTOH, it's going to be cheaper to do the sort, but there will never be that many bitmaps to sort, and it might put the wrong bitmap at the start... I think we'd need to see evidence this is a bottleneck to take the risk.

I made a throwaway comment in the RoaringBitmap repo a while ago thinking of this particular use case, but didn't do any kind of validation at the time

Yes Richard - we kept that PR on hold , as technicaly it will worsen the performance 

To take this to closure I did the benchmarkings - The result came out to be negative.

Even though cardinality calculation is expensive its better to sort by cardinality than container count. Will post the numbers by this weekend