Hi everyone, I am working on building compression...
# general
c
Hi everyone, I am working on building compression recommendations for our stakeholders using our Apache Pinot deployment. I am particularly interested in identifying the best compression methods for the following data types: 1. STRING 2. LONG/DOUBLE/FLOAT/DECIMAL 3. BOOLEAN 4. TIMESTAMP My team is working on an e-commerce database, and I expect most columns to have a high degree of cardinality. Does anyone have input or recommendations on the most effective compression methods for these data types? Thank you!
m
It is more of a function of cardinality than data type. For high cardinality (eg mostly unique values) use no-dict (or raw) index with v4 format and zstd compression. For low/med cardinality use dictionary encoding.
c
Thank you for replying Mayank! I will start by testing the compression types you've listed. In order to build our internal documentation I will need to do some proof of concept testing. Is there a list of supported compression methods anywhere? From searching slack I've seen: 1. gzip 2. snappy 3. lz4 4. zstd 5. dictionary (native)
Ah never mind i found it. https://docs.pinot.apache.org/basics/indexing/forward-index Scroll down to "The compressionCodec parameter has the following valid values:"