Hi, I have a parquet file of size 124 MB. The segment corresponding to it has size 10 GB with replication of 3. Are indexes inflating the size of the segment? Or what else could be the reason? There are no custom index configuration in the table config. We are using default settings for indexes.

{
  "tableName": "xxx",
  "tableType": "OFFLINE",
  "segmentsConfig": {
    "retentionTimeUnit": "DAYS",
    "retentionTimeValue": "365",
    "schemaName": "xxx",
    "replication": "3",
    "replicasPerPartition": "1",
    "segmentPushType": "APPEND",
    "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
    "timeColumnName": "time"
  },
  "ingestionConfig": {},
  "tenants": {},
  "fieldConfigList": [],
  "tableIndexConfig": {
    "loadMode": "MMAP"
  },
  "metadata": {
    "customConfigs": {}
  }
}

hi, are you able to get the segment metadata from the disk? it’s under pinot server data directory, you can find a file called

metadata.properties

Since pinot segment and parquet file is 1:1 mapping, we can compare your single parquet file size and corresponding pinot data size.

Thanks for the reply. This is what I found in metadata.properties

segment.padding.character = \u0000
segment.name = xxx_abd7d158e1d44cbfbd97aebc9b430156
segment.table.name = xxx
segment.dimension.column.names = col1,col2,col3
segment.datetime.column.names = processing_time
segment.time.column.name = processing_time
segment.total.docs = 4074079
segment.start.time = 1699346856829
segment.end.time = 1699346856829
segment.time.unit = MILLISECONDS
custom.input.data.file.uri = s3://bucket-path/xxx_abd7d158e1d44cbfbd97aebc9b430156.snappy.parquet
column.col1.cardinality = 4074079
column.col1.totalDocs = 4074079
column.col1.dataType = STRING
column.col1.bitsPerElement = 22
column.col1.lengthOfEachEntry = 15
column.col1.columnType = DIMENSION
column.col1.isSorted = false
column.col1.hasDictionary = true
column.col1.isSingleValues = true
column.col1.maxNumberOfMultiValues = 0
column.col1.totalNumberOfEntries = 4074079
column.col1.isAutoGenerated = false
column.col1.minValue = GEL900692970134
column.col1.maxValue = SVL999991299312
column.col1.defaultNullValue = null
column.col2.cardinality = 2956973
column.col2.totalDocs = 4074079
column.col2.dataType = STRING
column.col2.bitsPerElement = 22
column.col2.lengthOfEachEntry = 512
column.col2.columnType = DIMENSION
column.col2.isSorted = false
column.col2.hasDictionary = true
column.col2.isSingleValues = true
column.col2.maxNumberOfMultiValues = 0
column.col2.totalNumberOfEntries = 4074079
column.col2.isAutoGenerated = false
column.col2.minValue =
column.col2.maxValue = SAN999999999993
column.col2.defaultNullValue = null
column.processing_time.cardinality = 1
column.processing_time.totalDocs = 4074079
column.processing_time.dataType = TIMESTAMP
column.processing_time.bitsPerElement = 1
column.processing_time.lengthOfEachEntry = 0
column.processing_time.columnType = DATE_TIME
column.processing_time.isSorted = true
column.processing_time.hasDictionary = true
column.processing_time.isSingleValues = true
column.processing_time.maxNumberOfMultiValues = 0
column.processing_time.totalNumberOfEntries = 4074079
column.processing_time.isAutoGenerated = false
column.processing_time.datetimeFormat = 1:MILLISECONDS:EPOCH
column.processing_time.datetimeGranularity = 1:MILLISECONDS
column.processing_time.minValue = 1699346856829
column.processing_time.maxValue = 1699346856829
column.processing_time.defaultNullValue = 0
column.col3.cardinality = 4074079
column.col3.totalDocs = 4074079
column.col3.dataType = STRING
column.col3.bitsPerElement = 22
column.col3.lengthOfEachEntry = 512
column.col3.columnType = DIMENSION
column.col3.isSorted = false
column.col3.hasDictionary = true
column.col3.isSingleValues = true
column.col3.maxNumberOfMultiValues = 0
column.col3.totalNumberOfEntries = 4074079
column.col3.isAutoGenerated = false
column.col3.minValue = GEE900074558080
column.col3.maxValue = SVE999782599010
column.col3.defaultNullValue = null
segment.index.version = v3

ack, can you also paste an index_map file here

index_map

col1.dictionary.startOffset = 0
col1.dictionary.size = 61111193
col1.forward_index.startOffset = 61111193
col1.forward_index.size = 11203726
col2.dictionary.startOffset = 72314919
col2.dictionary.size = 1513970184
col2.forward_index.startOffset = 1586285103
col2.forward_index.size = 11203726
processing_time.dictionary.startOffset = 1597488829
processing_time.dictionary.size = 16
processing_time.forward_index.startOffset = 1597488845
processing_time.forward_index.size = 16
col3.dictionary.startOffset = 1597488861
col3.dictionary.size = 2085928456
col3.forward_index.startOffset = 3683417317
col3.forward_index.size = 11203726

yeah, col1 dict is 61M, while col2 is 1.5g, col3 is 2.1g

Thanks I will try it in the morning tomorrow and post the results here. So, is it basically the standard to use raw encoding when the cardinality of a column is high?

yes, Raw forward index is particularly effective for columns with a large number of unique values, where dictionary encoding doesn’t provide significant compression benefits.

Thanks for the help 🙂 I just disabled dictionary on these two columns and now the segment size is 179 MB which is close to the parquet file size of 124 MB. Sweet 🙂

"tableIndexConfig": {
    "loadMode": "MMAP",
    "noDictionaryColumns": [
      "col2",
      "col3"
   ]
  },

The configuration below however gave me an error while segment creation

"fieldConfigList": [
    {
      "name": "col2",
      "encodingType": "RAW",
      "indexes": {
        "forward": {
          "chunkCompressionType": "PASS_THROUGH",
          "deriveNumDocsPerChunk": false,
          "rawIndexWriterVersion": 2
        }
      }
    },

Caused by: org.apache.pinot.segment.spi.index.MergedColumnConfigDeserializer$ConfigDeclaredTwiceException: Configuration is declared in two different ways for column col2
        at org.apache.pinot.segment.spi.index.MergedColumnConfigDeserializer$OnConflict$1.merge(MergedColumnConfigDeserializer.java:78)
        at org.apache.pinot.segment.spi.index.MergedColumnConfigDeserializer.deserialize(MergedColumnConfigDeserializer.java:52)
        at org.apache.pinot.segment.spi.index.AbstractIndexType.getConfig(AbstractIndexType.java:65)
        at org.apache.pinot.segment.spi.index.FieldIndexConfigsUtil.readConfig(FieldIndexConfigsUtil.java:63)
        at org.apache.pinot.segment.spi.index.FieldIndexConfigsUtil.createIndexConfigsByColName(FieldIndexConfigsUtil.java:47)
        at org.apache.pinot.segment.spi.index.FieldIndexConfigsUtil.createIndexConfigsByColName(FieldIndexConfigsUtil.java:37)
        at org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig.<init>(SegmentGeneratorConfig.java:214)
        at org.apache.pinot.plugin.ingestion.batch.common.SegmentGenerationTaskRunner.run(SegmentGenerationTaskRunner.java:96)
        at org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner.lambda$submitSegmentGenTask$1(SegmentGenerationJobRunner.java:265)
        ... 5 more

seems conflict configs?

Is there any way to get the segment and index metadata through the sql or rest api? Seems it would be useful for people to see how space is used and see if they need to tune their config

there are segment metadata api, exposed from pinot controller

Another alternative is to use variable length dictionary. This will reduce the segment size, but also allow for inverted index, which no-dict column doesn’t allow for today