https://pinot.apache.org/ logo
Join Slack
Powered by
Another question: What kind of compression is used...
# general
n
Another question: What kind of compression is used for storing the column data in the segment files, is there any choice on the compression algorithm, It appears that column data is not compressed at all, because when I add a new column with LONG data type, the disk footprint increases by 10 bytes per record.
m
By default there’s a dictionary encoding that provides compression. Based on your description I am guessing it is a very high cardinality column? Can you share the schema and the metadata.properties + index_map files?
n
Here oid is the high cardinality column (6Million+)
schema2.json,schema2_metadata.properties,schema2_index_map,schema1.json,schema1_metadata.properties,schema1_index_map
schema2.jsonschema1_metadata.propertiesschema2_metadata.propertiesschema2_index_mapschema1.jsonschema1_index_map
I took a segment file and compressed it using gzip, the compression ratio is 2.6, this means the data is not compressed optimally. Can we expect some kind of enhancement in pinot where it can use a better compression technique on the segment file or serve the segment from a compressed file.
Snappy compression on the same data yields a compression ratio of 1.7
When I try to change the chunkCompressionType to SNAPPY, pinot ignores the changes and goes back to default.
Copy code
"fieldConfigList": [
      {
        "name": "value",
        "encodingType": "RAW",
        "indexes": {
          "forward": {
            "chunkCompressionType": "SNAPPY",
            "deriveNumDocsPerChunk": false,
            "rawIndexWriterVersion": 2
          }
        },
        "tierOverwrites": null
      }
    ],
Open in Slack
PreviousNext
Powered by