Kishore G
Kishore G
Neha Pawar
Kishore G
Neha Pawar
Dan Hill
06/26/2020, 5:25 PMinsertions
is just a long metric column. It's not a dimension. I'm not sure why it's being hitting this.
Metrics aggregation cannot be turned ON in presence of dictionary encoded metrics, eg: insertions
Sidd
06/26/2020, 5:26 PMMayank
Mayank
Sidd
06/26/2020, 5:31 PMKishore G
Dan Hill
06/26/2020, 5:35 PMSidd
06/26/2020, 5:36 PMDan Hill
06/26/2020, 5:37 PMDan Hill
06/26/2020, 5:44 PMpinot-server-0
log is showing that my params in streamConfig
were supplied but isn't a known config
. Any ideas?
2020/06/26 17:26:05.729 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.hlc.zk.connect.string' was supplied but isn't a known config.
2020/06/26 17:26:05.729 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'realtime.segment.flush.threshold.size' was supplied but isn't a known config.
2020/06/26 17:26:05.737 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'stream.kafka.decoder.class.name' was supplied but isn't a known config.
2020/06/26 17:26:05.737 WARN [ConsumerConfig] [HelixTaskExecutor-message_handle_thread] The configuration 'streamType' was supplied but isn't a known config.
...
Here is what my table config looks like (with some repetitive parts removed)
{
"tableName": "metrics",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "timestamp",
"timeType": "MILLISECONDS",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "5",
"segmentPushType": "APPEND",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
"schemaName": "metrics",
"replication": "1",
"replicasPerPartition": "1"
},
"tableIndexConfig": {
"loadMode" : "MMAP",
"starTreeIndexConfigs": [
{
"dimensionsSplitOrder": [
"platform_id",
... my other dimensions
],
"skipStarNodeCreationForDimensions": [
],
"functionColumnPairs": [
"SUM__insertions",
... my other metrics
]
}
],
"streamConfigs": {
"streamType": "kafka",
"stream.kafka.consumer.type": "simple",
"stream.kafka.topic.name": "metrics-realtime",
... all of the configs are ignored
}
},
"tenants": {},
"metadata": {
"customConfigs": {}
}
}
Neha Pawar
Kishore G
Dan Hill
06/26/2020, 5:52 PMPradeep
06/26/2020, 8:47 PMselect * from <tablename> order by timestamp limit 10
returns empty.
I also see some of the old segments to be in error state from the broker logs
Resource: <tablename>_REALTIME, partition: <tablename>__7__0__20200625T1913Z is in ERROR state
Resource: <tablename>_REALTIME, partition: <tablename>__8__0__20200625T1913Z is in ERROR state
Resource: <tablename>_REALTIME, partition: <tablename>__9__0__20200625T1913Z is in ERROR state
Pradeep
06/26/2020, 8:47 PMMayank
Pradeep
06/26/2020, 8:49 PM<tablename>__9__0__20200625T1913Z
is one of the segment that went into error state (though it’s an older segment)Pradeep
06/26/2020, 8:49 PMMayank
Mayank
Pradeep
06/26/2020, 8:55 PMMayank
Pradeep
06/26/2020, 8:57 PMPradeep
06/26/2020, 9:05 PMSleep for 10000ms as service status has not turned GOOD: IdealStateAndCurrentStateMatchServiceStatusCallback:partition=<tablename>__57__0__20200625T191
3Z, expected=ONLINE, found=OFFLINE, creationTime=1593202611703, modifiedTime=1593202650032, version=95, waitingFor=CurrentStateMatch, resource=<tablename>_REALTIME, numResourcesLeft=1, numTotalResources=1, minStartCount=1,;IdealStateAndExternalViewMatchServiceStatusCallback:Init;
Caught exception in state transition from OFFLINE -> ONLINE for resource: <tablename>_REALTIME, partition: <tablename>__57__0__20200625T1913Z
java.lang.RuntimeException: org.apache.pinot.spi.utils.retry.AttemptsExceededException: Operation failed after 3 attempts
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.downloadAndReplaceSegment(RealtimeTableDataManager.java:286) ~[pinot-all-0
.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.core.data.manager.realtime.RealtimeTableDataManager.addSegment(RealtimeTableDataManager.java:252) ~[pinot-all-0.4.0-jar-with-d
ependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.server.starter.helix.HelixInstanceDataManager.addRealtimeSegment(HelixInstanceDataManager.java:132) ~[pinot-all-0.4.0-jar-with
-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at org.apache.pinot.server.starter.helix.SegmentOnlineOfflineStateModelFactory$SegmentOnlineOfflineStateModel.onBecomeOnlineFromOffline(SegmentOnlineOfflineStateModelFactory.java:164) [pinot-all-0.4.0-jar-with-dependencies.jar:0.4.0-8355d2e0e489a8d127f2e32793671fba505628a8]
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) ~[?:?]
Kishore G