Hi team I ve an issue using timestamp index Any idea how to Apache Pinot #troubleshooting

Hi team, I’ve an issue using timestamp index. Any ...

Alice

07/21/2022, 1:33 AM

Hi team, I’ve an issue using timestamp index. Any idea how to solve it? I’m using pinot 0.11 master and testing timestamp index. The first batch of segments failed building due to the following error. Then the table stopped consuming stream data.

Copy code

2022/07/20 08:05:35.579 ERROR [LLRealtimeSegmentDataManager_table_name_stage__1__0__20220720T0805Z] [table_name_stage__1__0__20220720T0805Z] Could not build segment
java.lang.NullPointerException: null
	at org.apache.pinot.segment.spi.creator.ColumnIndexCreationInfo.getDistinctValueCount(ColumnIndexCreationInfo.java:67) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.init(SegmentColumnarIndexCreator.java:201) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:216) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:123) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:840) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:767) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:666) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at java.lang.Thread.run(Thread.java:829) [?:?]

Alice

07/21/2022, 1:34 AM

Copy code

Here's the timestamp schema and index config:
"dateTimeFieldSpecs": [
    {
      "name": "timestamps",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }
]
"fieldConfigList": [
      {
        "name": "timestamps",
        "encodingType": "DICTIONARY",
        "indexType": "TIMESTAMP",
        "indexTypes": [
          "TIMESTAMP"
        ],
        "timestampConfig": {
          "granularities": [
            "HOUR",
            "DAY",
            "WEEK",
            "MONTH"
          ]
        }
      }
    ],

Xiang Fu

07/21/2022, 2:42 AM

So it works without timestamp index?

Xiang Fu

07/21/2022, 2:42 AM

I will take a look

Alice

07/21/2022, 2:51 AM

Works well without timestamp index.

Mayank

07/21/2022, 4:54 AM

@Jackie ^^

Alice

07/21/2022, 7:49 AM

Hey, @Xiang Fu I added more fields in schema and it worked. I thought Pinot will created these fields config by default, since no doc pointed it out. I’m not sure it’s the right way to add this config, though it did solved this issue.😅

Copy code

{
      "name": "timestamps",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    },
    {
      "name": "$timestamps$HOUR",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    },
    {
      "name": "$timestamps$DAY",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    },
    {
      "name": "$timestamps$WEEK",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    },
    {
      "name": "$timestamps$MONTH",
      "dataType": "TIMESTAMP",
      "format": "1:MILLISECONDS:EPOCH",
      "granularity": "1:MILLISECONDS"
    }

Xiang Fu

07/21/2022, 7:56 AM

hmm, interesting, those columns should be created automatically, I will double check that !

Xiang Fu

07/21/2022, 7:56 AM

Thanks for the hints!

😄 1

Open in Slack

Previous Next