Georgi Varbanov
05/12/2025, 11:57 AMGeorgi Varbanov
05/12/2025, 11:58 AMGeorgi Varbanov
05/12/2025, 11:59 AMGeorgi Varbanov
05/12/2025, 12:04 PMGeorgi Varbanov
05/12/2025, 12:05 PMGeorgi Varbanov
05/12/2025, 12:06 PMGeorgi Varbanov
05/12/2025, 12:44 PMGeorgi Varbanov
05/12/2025, 12:47 PMGeorgi Varbanov
05/12/2025, 12:47 PMGeorgi Varbanov
05/12/2025, 12:54 PMGeorgi Varbanov
05/12/2025, 12:55 PMXiang Fu
Georgi Varbanov
05/13/2025, 6:35 AMapiVersion: v1
kind: ConfigMap
metadata:
name: pinot-server-config
labels:
app: pinot
component: server
data:
pinot-server.conf: |-
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/var/pinot/server/data/index
pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
pinot.set.instance.id.to.hostname=true
pinot.server.instance.realtime.alloc.offheap=true
pinot.query.server.port=7321
pinot.query.runner.port=7732
realtime.segment.serverUploadToDeepStore = true
pinot.server.instance.segment.stream.download.untar=true
pinot.server.instance.segment.stream.download.untar.rate.limit.bytes.per.sec=100000000
pinot.server.instance.table.level.max.parallel.segment.downloads=4
pinot.server.instance.enable.split.commit=true
pinot.server.netty.native.transports.enabled=trueGeorgi Varbanov
05/13/2025, 6:36 AMGeorgi Varbanov
05/13/2025, 6:37 AMapiVersion: v1
kind: ConfigMap
metadata:
name: pinot-controller-config
labels:
app: pinot
component: controller
data:
pinot-controller.conf: |-
controller.port=9000
pinot.set.instance.id.to.hostname=true
# Cluster name
controller.helix.cluster.name=pinot-quickstart
# ZooKeeper configuration
controller.zk.str=pinot-zookeeper:2181
# External access
controller.access.protocols.http.port=9000
# Controller periodic tasks
controller.task.scheduler.enabled=true
controller.task.frequencyPeriod=5m
# System resources
controller.data.dir=/var/pinot/controller/data
controller.local.temp.dir=/tmp/pinot-controller-tmp
controller.deleted.segments.retentionInDays=1
# Optional: Throttling config for high volume
controller.realtime.segment.commit.timeoutMs=120000
# tuning
controller.allow.hlc.tables=false
controller.enable.split.commit=trueGeorgi Varbanov
05/13/2025, 6:51 AMXiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 6:56 AMXiang Fu
Xiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 7:00 AMXiang Fu
Georgi Varbanov
05/13/2025, 7:02 AMXiang Fu
Xiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 7:03 AMXiang Fu
Georgi Varbanov
05/13/2025, 7:03 AMGeorgi Varbanov
05/13/2025, 7:04 AMXiang Fu
Georgi Varbanov
05/13/2025, 7:04 AMGeorgi Varbanov
05/13/2025, 7:05 AMXiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 7:09 AMXiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 7:11 AMGeorgi Varbanov
05/13/2025, 7:12 AMGeorgi Varbanov
05/13/2025, 7:12 AMGeorgi Varbanov
05/13/2025, 7:12 AMXiang Fu
Xiang Fu
Xiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 7:15 AMGeorgi Varbanov
05/13/2025, 7:16 AMXiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 7:24 AM{
"segment.realtime.endOffset": "3405878",
"segment.start.time": "1534184236957",
"segment.time.unit": "MILLISECONDS",
"segment.flush.threshold.size": "100000",
"segment.realtime.startOffset": "3377687",
"segment.size.in.bytes": "24639413",
"segment.end.time": "1579261376287",
"segment.total.docs": "100004",
"segment.realtime.numReplicas": "3",
"segment.creation.time": "1746706039300",
"segment.index.version": "v3",
"segment.crc": "4258377168",
"segment.partition.metadata": "{\"columnPartitionMap\":{\"customer_id\":{\"numPartitions\":100,\"functionName\":\"Murmur\",\"partitions\":[0,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,21,25,27,28,29,32,33,36,37,39,40,41,43,44,46,47,49,50,51,52,53,54,55,59,60,61,62,63,64,65,66,68,70,72,73,74,76,77,79,80,81,82,84,85,86,87,90,91,92,93,94,95,98,99],\"functionConfig\":null}}}",
"segment.realtime.status": "DONE",
"segment.download.url": "http://pinot-controller-0.pinot-controller.pinot.svc.cluster.local:9000/segments/bets/bets__0__0__20250508T1207Z"
}Georgi Varbanov
05/13/2025, 7:27 AMXiang Fu
realtime.segment.flush.threshold.size is a deprecated config, you can remove it then it should be -1 as defaultXiang Fu
5_000_000Xiang Fu
0 then pinot should auto adjust the flush size, can you check latest a few segments metadta?Xiang Fu
bets__0__100__202505xxxxxZGeorgi Varbanov
05/13/2025, 9:55 AMXiang Fu
"realtime.segment.flush.threshold.segment.rows": "0",
"realtime.segment.flush.threshold.time": "24h",
"realtime.segment.flush.threshold.segment.size": "500M",
"realtime.segment.flush.threshold.size": "0"
initially for the table flush config?Georgi Varbanov
05/13/2025, 10:48 AMGeorgi Varbanov
05/13/2025, 10:49 AMXiang Fu
Georgi Varbanov
05/13/2025, 10:50 AMGeorgi Varbanov
05/13/2025, 10:50 AMXiang Fu
completionMode to Download . This will avoid the local segment build which may take long timeXiang Fu
Georgi Varbanov
05/13/2025, 10:51 AMXiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 10:51 AMGeorgi Varbanov
05/13/2025, 10:52 AMGeorgi Varbanov
05/13/2025, 10:52 AMXiang Fu
Georgi Varbanov
05/13/2025, 10:52 AMXiang Fu
Georgi Varbanov
05/13/2025, 10:54 AMGeorgi Varbanov
05/13/2025, 10:55 AMGeorgi Varbanov
05/13/2025, 10:55 AMGeorgi Varbanov
05/13/2025, 10:56 AMXiang Fu
Georgi Varbanov
05/13/2025, 11:22 AMGeorgi Varbanov
05/13/2025, 11:36 AMGeorgi Varbanov
05/13/2025, 11:37 AMXiang Fu
Georgi Varbanov
05/13/2025, 11:57 AMXiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 12:00 PMGeorgi Varbanov
05/13/2025, 12:01 PMGeorgi Varbanov
05/13/2025, 12:01 PMGeorgi Varbanov
05/13/2025, 12:02 PMXiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 12:04 PMXiang Fu
Georgi Varbanov
05/13/2025, 12:05 PMGeorgi Varbanov
05/13/2025, 12:05 PMXiang Fu
Georgi Varbanov
05/13/2025, 12:05 PMGeorgi Varbanov
05/13/2025, 12:06 PMXiang Fu
toEpochDays will divide epoch millis to epoch day valueXiang Fu
Xiang Fu
1747138079000 is millis epoch, the days epoch value is 20221, then convert it back to 1747094400000Georgi Varbanov
05/13/2025, 12:09 PMXiang Fu
fromEpochDays(toEpochDays(creation_date)) as the transform functinoGeorgi Varbanov
05/13/2025, 12:11 PMGeorgi Varbanov
05/13/2025, 12:11 PMGeorgi Varbanov
05/13/2025, 12:17 PMGeorgi Varbanov
05/13/2025, 12:18 PMXiang Fu
Xiang Fu
Xiang Fu
Georgi Varbanov
05/13/2025, 12:22 PM