Apache Pinot

Is it possible to use LZ4 compression type for metric columns?

If you use no-dictionary indexing on them, you can. May I ask what’s the use case. cc: <@U02DXKP9B6C>

Yes, we did put the metric in the no dictionary columns but it seems in the realtime segment converter it still does passthrough compression type for metric columns..

<https://github.com/apache/pinot/blob/f2f8e38f9424bcacf3946197c9afcd50ef1d58fa/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java#L100|https://github.com/apache/pinot/blob/f2f8e38f9424bcacf3946197c9afcd50ef1d58fa/pino[…]/segment/local/realtime/converter/RealtimeSegmentConverter.java>

<https://github.com/apache/pinot/issues/7973>

as mentioned on the issue, LZ4 is unlikely to be a win for numeric columns. Encoding (e.g. delta, xor, pfor, gorilla) has been on the cards for a while but nobody has got round to it. It could be added soon depending on prioritisation.