https://pinot.apache.org/ logo
Join Slack
Powered by
Hi team, if a segment failed building, the segment...
# troubleshooting
a
Hi team, if a segment failed building, the segment status is ‘NOT_CONSUMING’, but the table status is GOOD. Through pinot server log, I see offset is forwarding, when data is kept being written to Kafka. However, data isn’t really ingested in pinot server because data couldn’t be queried and the table row number isn’t increasing. This would lead to data loss using segment reset api. My question is is there any way to avoid data loss?
s
What is the segment state in IDEALSTATE and EXTERNALVIEW?
s
If the segment is not built, there must be an exception logged. Can you get the exception please?
a
Yes, here’s the error info.
Copy code
2022/07/20 08:05:35.579 ERROR [LLRealtimeSegmentDataManager_table_name_stage__1__0__20220720T0805Z] [table_name_stage__1__0__20220720T0805Z] Could not build segment
java.lang.NullPointerException: null
	at org.apache.pinot.segment.spi.creator.ColumnIndexCreationInfo.getDistinctValueCount(ColumnIndexCreationInfo.java:67) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator.init(SegmentColumnarIndexCreator.java:201) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:216) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:123) ~[pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:840) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:767) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:666) [pinot-all-0.11.0-SNAPSHOT-jar-with-dependencies.jar:0.11.0-SNAPSHOT-4f717514068d54784f4636ca9e07dd00521e8d86]
	at java.lang.Thread.run(Thread.java:829) [?:?]
s
Hmm...this does not seem very useful. columnStatistics is somehow becoming null, I believe. I am not sure how this can happen.
a
Copy code
I got another segment build error and had the same issue. Here's the error.
2022/07/16 14:46:48.394 INFO [ImmutableSegmentImpl] [table_name__1__15__20220716T0523Z] Trying to destroy segment : table_name__1__15__20220716T0523Z
2022/07/16 14:46:48.395 ERROR [LLRealtimeSegmentDataManager_table_name__1__15__20220716T0523Z] [table_name__1__15__20220716T0523Z] Could not build segment
java.lang.IllegalArgumentException: Cannot add NaN to t-digest
	at com.tdunning.math.stats.MergingDigest.add(MergingDigest.java:199) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at com.tdunning.math.stats.MergingDigest.add(MergingDigest.java:194) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at com.tdunning.math.stats.AbstractTDigest.add(AbstractTDigest.java:131) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.aggregator.PercentileTDigestValueAggregator.getInitialAggregatedValue(PercentileTDigestValueAggregator.java:54) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.aggregator.PercentileTDigestValueAggregator.getInitialAggregatedValue(PercentileTDigestValueAggregator.java:27) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.mergeSegmentRecord(BaseSingleTreeBuilder.java:260) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.startree.v2.builder.OffHeapSingleTreeBuilder$1.next(OffHeapSingleTreeBuilder.java:254) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.startree.v2.builder.OffHeapSingleTreeBuilder$1.next(OffHeapSingleTreeBuilder.java:242) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.startree.v2.builder.BaseSingleTreeBuilder.build(BaseSingleTreeBuilder.java:306) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.startree.v2.builder.OffHeapSingleTreeBuilder.build(OffHeapSingleTreeBuilder.java:42) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.startree.v2.builder.MultipleTreesBuilder.build(MultipleTreesBuilder.java:141) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.buildStarTreeV2IfNecessary(SegmentIndexCreationDriverImpl.java:328) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:292) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:240) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:123) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:840) [pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:767) [pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:666) [pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
	at java.lang.Thread.run(Thread.java:829) [?:?]
	Suppressed: java.lang.NullPointerException
		at org.apache.pinot.segment.local.startree.v2.builder.OffHeapSingleTreeBuilder.close(OffHeapSingleTreeBuilder.java:346) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at org.apache.pinot.segment.local.startree.v2.builder.MultipleTreesBuilder.build(MultipleTreesBuilder.java:139) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.buildStarTreeV2IfNecessary(SegmentIndexCreationDriverImpl.java:328) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:292) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:240) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at org.apache.pinot.segment.local.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:123) ~[pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:840) [pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:767) [pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:666) [pinot-all-my.jar:0.11.0-SNAPSHOT-ca07ee568b94513f9da40b2b320a17317558eb23]
		at java.lang.Thread.run(Thread.java:829) [?:?]
2022/07/16 14:46:48.396 ERROR [LLRealtimeSegmentDataManager_table_name__1__15__20220716T0523Z] [table_name__1__15__20220716T0523Z] Could not build segment for table_name__1__15__20220716T0523Z
Open in Slack
PreviousNext
Powered by