Timothy James
08/19/2022, 12:32 AM"task": {
"taskTypeConfigsMap": {
"MergeRollupTask": {
"5min.bucketTimePeriod": "5min",
"5min.bufferTimePeriod": "15min",
"1hour.bucketTimePeriod": "1h",
"1hour.bufferTimePeriod": "2h",
"1day.bucketTimePeriod": "1d",
"1day.bufferTimePeriod": "1d"
}
}
},
Timothy James
08/19/2022, 12:33 AM{
"OFFLINE": {
"tableName": "simple_OFFLINE",
"tableType": "OFFLINE",
"segmentsConfig": {
"schemaName": "simple",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "1",
"replication": "1",
"segmentPushType": "APPEND",
"segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy"
},
"tenants": {
"broker": "DefaultTenant",
"server": "DefaultTenant"
},
"tableIndexConfig": {
"invertedIndexColumns": [
"a",
"b"
],
"rangeIndexVersion": 2,
"autoGeneratedInvertedIndex": false,
"createInvertedIndexDuringSegmentGeneration": false,
"loadMode": "HEAP",
"enableDefaultStarTree": false,
"enableDynamicStarTreeCreation": false,
"aggregateMetrics": false,
"nullHandlingEnabled": false
},
"metadata": {
"customConfigs": {}
},
"task": {
"taskTypeConfigsMap": {
"MergeRollupTask": {
"5min.bucketTimePeriod": "5min",
"5min.bufferTimePeriod": "15min",
"1hour.bucketTimePeriod": "1h",
"1hour.bufferTimePeriod": "2h",
"1day.bucketTimePeriod": "1d",
"1day.bufferTimePeriod": "1d"
}
}
},
"fieldConfigList": [],
"isDimTable": false
}
}
Schema:
{
"schemaName": "simple",
"dimensionFieldSpecs": [
{
"name": "a",
"dataType": "STRING"
},
{
"name": "b",
"dataType": "STRING"
}
]
}
controller config map, per docs, contains:
# For segment merges:
controller.task.scheduler.enabled=true
controller.task.frequencyPeriod=5m
Timothy James
08/19/2022, 12:34 AMTimothy James
08/19/2022, 12:36 AMTimothy James
08/22/2022, 8:03 PMpinot-quickstart
from https://docs.pinot.apache.org/basics/getting-started/kubernetes-quickstart .
I'm assuming that since the result of this (which uses a Helm chart) includes running minion pods that everything needed for this merge task is supported by the version of Pinot I'm using — whatever version those instructions happen to use, which I don't see in the controller web UI anywhere. Is that a valid assumption?Mayank
Haitao Zhang
08/23/2022, 9:05 PMHaitao Zhang
08/24/2022, 7:01 PM5m
2. we should add a time column to ensure MergeRollup can take filter segments into different time bucketsMayank
Timothy James
08/24/2022, 7:03 PM1. "5min.bucketTimePeriod": "5min" should beAlready fixed.5m
2. we should add a time column to ensure MergeRollup can take filter segments into different time bucketsWorking on it! The MergeRollup task docs were not, I think, clear on this one, and could be made so. Thanks!