https://pinot.apache.org/ logo
Join Slack
Powered by
Hi We are using REALTIME table in pinot to ingest ...
# general
r
Hi We are using REALTIME table in pinot to ingest data from a Kinesis stream. But we are not seeing any segment data in S3. We have already gave full access to S3 for controller/broker/server EC2 nodes. We already validated that pinot processes can reach S3 - so no access/network issues possible, Here is the Controller config # Location to store Pinot Segments pushed from clients`pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS` 
controller.local.temp.dir=/tmp/pinot-tmp-data/
pinot.controller.storage.factory.s3.region={region}
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.controller.storage.factory.s3.disableAcl=false
controller.data.dir=s3://{bucketName}/pinot-data/pinot-s3-example/controller-data
pinot table config 
{
"tableName": "kinesisTable",
"tableType": "REALTIME",
"segmentsConfig": {
"timeColumnName": "creationTimestamp",
"replicasPerPartition": "1",
"replication": "1",
"retentionTimeUnit": "DAYS",
"retentionTimeValue": "7"
},
"tenants": {},
"tableIndexConfig": {
"loadMode": "MMAP",
"streamConfigs": {
"streamType": "kinesis",
"stream.kinesis.topic.name": "pinot-stream",
"region": "{AWSRegion}",
"shardIteratorType": "LATEST",
"stream.kinesis.consumer.type": "lowlevel",
"stream.kinesis.fetch.timeout.millis": "30000",
"stream.kinesis.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
"stream.kinesis.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kinesis.KinesisConsumerFactory",
"realtime.segment.flush.threshold.rows": "0",
"realtime.segment.flush.threshold.time": "1h",
"realtime.segment.flush.threshold.size": "150M"
}
},
"metadata": {
"customConfigs": {}
}
}
x
How long have you wait ? The table flush threshold is 1 hour, so you may need to wait for a while
Once realtime segment is flushed, it should be persisted on s3. You can also check the segment metadata for segment download url
r
@Xiang Fu I already waited for many hours - still not seeing it. "`You can also check the segment metadata for segment download url`" How would I see it?
x
go to table view, you should see a list of segments on bottom left
then click segments not in CONSUMING status
those are persisted segments.
r
@Xiang Fu All 4 segments are in consuming status But I have the flush threshold as 1 hour. Shouldn't that create the segments every hour and push it to S3? 
"realtime.segment.flush.threshold.time": "1h",
Metadata for the 1st segment 
{
"segment.realtime.numReplicas": "1",
"segment.creation.time": "1707771628058",
"segment.flush.threshold.size": "100000",
"segment.realtime.startOffset": "{\"shardId-000000000000\":\"49649210344722884761651303877664502814676963670236332034\"}",
"segment.realtime.status": "IN_PROGRESS"
}
https://startree.ai/blog/best-practices-for-designing-tables-in-apache-pinot This is not happening - do we have a bug? @Mayank
Copy code
By default, segments continue to consume data until the configured segment flush threshold time (default 6h) or segment flush threshold size (default 100 MB), whichever occurs first, is reached. At that point, the segment gets flushed out from memory to deepstore.
x
From the log, seems you haven’t set a time based flush threshold?
r
x
oh, can you set threshold rows to not 0 value , maybe something like 1000000?
r
Even though I set zero in the table definition, segment metadata says segment.flush.threshold.size = 100,000. It almost look like table config value specified is not getting used
x
right, when you set the threshold to 0. Pinot will auto adjust it for you.
Anyway this shouldn’t be a problem when you set the flush threshold to 1h
Can you search server log for the flush messages?
r
do you know the location on the controller log?
Changing the threshold to 50 did not make any difference either. Here is the controller log
pinot-all.log
x
hmm
Can you check the controller log for this
r
^^ that was the controller log
server conf: per https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot 
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/tmp/pinot-tmp/server/index
pinot.server.instance.segmentTarDir=/tmp/pinot-tmp/server/segmentTars
pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region={region}
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
controller conf: per doc 
# Pinot Role
pinot.service.role=CONTROLLER
# Pinot Cluster name
pinot.cluster.name=pinot-quickstart
# Pinot Zookeeper Server
controller.zk.str={zookeeperIP}
# Use hostname as Pinot Instance ID other than IP
<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true
# Pinot Controller Port
controller.port=9000
# Pinot Controller VIP Host
controller.vip.host={LoadBalancerDNS}
# Pinot Controller VIP Port
controller.vip.port=80
# Location to store Pinot Segments pushed from clients
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
controller.local.temp.dir=/tmp/pinot-tmp-data/
pinot.controller.storage.factory.s3.region={region}
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.controller.storage.factory.s3.disableAcl=false
controller.data.dir=s3://{bucketName}/pinot-data/pinot-s3-example/controller-data
@Xiang Fu S3 config looks correct on both controller/server. Also SSH'ed into both and verified S3 connectivity as well
Confirmed consumed segment status as OFFLINE - can see the segment data in server but not on S3 
{
"segmentName": "kinesisTable__0__0__20240213T1615Z",
"serverState": {
"Server_10.0.3.194_8098": {
"idealState": "OFFLINE",
"externalView": "OFFLINE",
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
}
}
}
[root@ip-10-0-3-194 kinesisTable_REALTIME]# ls -alt
total 388
drwxr-xr-x 2 root root     50 Feb 13 18:14 consumers
-rw-r--r-- 1 root root   1089 Feb 13 17:20 segment-stats.ser
drwxr-xr-x 6 root root    162 Feb 13 17:20 .
-rw-r--r-- 1 root root 392420 Feb 13 17:15 kinesisTable__0__0__20240213T1615Z.tar.gz
drwxr-xr-x 2 root root      6 Feb 13 17:15 _tmp
drwxr-xr-x 3 root root     16 Feb 13 17:15 kinesisTable__0__0__20240213T1615Z
drwxr-xr-x 3 root root     35 Feb 13 16:15 ..
drwxr-xr-x 2 root root      6 Feb 13 16:15 tmp
^^^ offline segment from server local disk
2 Views
Open in Slack
PreviousNext
Powered by