It seems like “realtime.segment.flush.threshold.ti...
# troubleshootingy
Yeongju Kang
11/05/2021, 7:23 AMIt seems like “realtime.segment.flush.threshold.time” doesn’t work. I set it as 30m and put some data and waited, but its segment size didn’t increase.
My config location is REALTIME > tableIndexConfig > streamConfigs > “realtime.segment.flush.threshold.time”: “30m”
<UPDATE>
FYI, 1h config works.
Is there something like floor value internally?
j
jagadesh
11/05/2021, 9:59 AM@Yeongju Kang Tried checking the code. TimeUtils working perfectly for both
30m1hk
Kishore G
11/05/2021, 2:47 PM
Do you have any other config?
m
Mayank
11/08/2021, 11:25 PM
Also @Yeongju Kang, could you elaborate the reasoning behind using a small time window? If your event rate is not large enough, you might end up getting too many small segments (not recommended).
y
Yeongju Kang
11/09/2021, 1:40 AM@Mayank Oh I will not configure with small value, I used this short flush period to do schema evolution.
Yeongju Kang
11/09/2021, 1:49 AMsegments in the picture are sequential
and my config is like this.
after this 1h30m period, working fine.
I set kafka consuming option as smallest and there was massive consumption around the flushing time. Could it affect?
{
“REALTIME”: {
“tableName”: “transcript2_with_time_flush_REALTIME”,
“tableType”: “REALTIME”,
“segmentsConfig”: {
“timeType”: “SECONDS”,
“schemaName”: “transcript”,
“timeColumnName”: “timestamp”,
“allowNullTimeValue”: false,
“replicasPerPartition”: “2”
},
“tenants”: {
“broker”: “DefaultTenant”,
“server”: “DefaultTenant”
},
“tableIndexConfig”: {
“rangeIndexVersion”: 1,
“autoGeneratedInvertedIndex”: false,
“createInvertedIndexDuringSegmentGeneration”: false,
“loadMode”: “MMAP”,
“streamConfigs”: {
“streamType”: “kafka”,
“stream.kafka.consumer.type”: “lowlevel”,
“stream.kafka.topic.name”: “transcript”,
“stream.kafka.decoder.class.name”: “org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder”,
“stream.kafka.consumer.factory.class.name”: “org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory”,
“stream.kafka.broker.list”: “b-1.kafka-engine-v2.robgzv.c2.kafka.ap-northeast-2.amazonaws.com:9092”,
“stream.kafka.consumer.prop.auto.offset.reset”: “smallest”,
“realtime.segment.flush.threshold.time”: “30m”
},
“enableDefaultStarTree”: false,
“enableDynamicStarTreeCreation”: false,
“aggregateMetrics”: false,
“nullHandlingEnabled”: false
},
“metadata”: {
“customConfigs”: {}
},
“routing”: {
“instanceSelectorType”: “strictReplicaGroup”
},
“upsertConfig”: {
“mode”: “FULL”,
“hashFunction”: “NONE”
},
“isDimTable”: false
}
}