Hi Guys, We are tring to set a cold storage on ou...
# troubleshooting
l
Hi Guys, We are tring to set a cold storage on our pinot cluster backed up by S3. In order to setup hot/cold storage for Pinot, and use S3 for cold storage the configurationsfor server.conf, controller.conf, broker.conf are attached at the end as well as the used schema and table config. We observed that the controler uses the S3 configuration as expected (controller.data.dir=s3://storage/controller) and uses the identified bucket as storage. But the server created a local folder that creates the following path: /s3:/cold-storage/server/tiered_REALTIME If anyone has any comments or any help on how we can fix this issue since we are uable to have the sgments moving to S3. Server: pinot.zk.server=localhost:2191 server.helix.cluster.name=PinotCluster pinot.server.netty.port=18098 pinot.server.netty.host=localhost pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.server.storage.factory.s3.endpoint=http://localhost:9000 pinot.server.storage.factory.s3.accessKey=minioadmin pinot.server.storage.factory.s3.secretKey=minioadmin pinot.server.storage.factory.s3.region=us-east-1 pinot.server.storage.factory.s3.enableS3A=false pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.server.segment.fetcher.protocols=file,http,s3 pinot.server.instance.tierConfigs.tierNames=hotTier,coldTier pinot.server.instance.segment.directory.loader=tierBased pinot.server.instance.dataDir=/Shared/pinot_data/server pinot.server.instance.tierConfigs.hotTier.dataDir=s3://hot-storage/server pinot.server.instance.tierConfigs.coldTier.dataDir=s3://cold-storage/server Controller: pinot.zk.server=localhost:2191 controller.helix.cluster.name=PinotCluster controller.port=19000 controller.host=localhost controller.tls.client.auth=false controller.segment.relocator.frequencyPeriod=60s controller.segmentRelocator.initialDelayInSeconds=10 controller.segmentRelocator.enableLocalTierMigration=true controller.enable.split.commit=true pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS pinot.controller.storage.factory.s3.endpoint=http://localhost:9000 pinot.controller.storage.factory.s3.accessKey=minioadmin pinot.controller.storage.factory.s3.secretKey=minioadmin pinot.controller.storage.factory.s3.region=us-east-1 pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher pinot.controller.segment.fetcher.protocols=file,http,s3 controller.data.dir=s3://storage/controller controller.local.temp.dir=/Shared/pinot_data/controller/temp Broker: pinot.zk.server=localhost:2191 broker.helix.cluster.name=PinotCluster broker.helix.port=18099 pinot.broker.hostname=localhost pinot.broker.client.queryPort=18099 table_config:
Copy code
{
  "tableName": "tiered",
  "tableType": "REALTIME",
  "segmentsConfig": {
    "minimizeDataMovement": false,
    "timeColumnName": "timestamp",
    "timeType": "MILLISECONDS",
    "replicasPerPartition": "1",
    "schemaName": "tiered",
    "replication": "2"
  },
  "tenants": {
    "broker": "DefaultTenant",
    "server": "DefaultTenant",
    "tagOverrideConfig": {}
  },
  "tableIndexConfig": {
    "autoGeneratedInvertedIndex": false,
    "createInvertedIndexDuringSegmentGeneration": false,
    "loadMode": "MMAP",
    "streamConfigs": {
      "streamType": "kafka",
      "stream.kafka.topic.name": "tiered",
      "stream.kafka.broker.list": "localhost:19092",
      "stream.kafka.consumer.type": "lowlevel",
      "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
      "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
      "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
      "realtime.segment.flush.threshold.rows": "0",
      "realtime.segment.flush.threshold.segment.rows": "0",
      "realtime.segment.flush.threshold.time": "1m",
      "realtime.segment.flush.threshold.segment.size": "100M"
    },
    "enableDefaultStarTree": false,
    "enableDynamicStarTreeCreation": false,
    "aggregateMetrics": false,
    "nullHandlingEnabled": false,
    "columnMajorSegmentBuilderEnabled": true,
    "optimizeDictionary": false,
    "optimizeDictionaryForMetrics": false,
    "optimizeDictionaryType": false,
    "noDictionarySizeRatioThreshold": 0.85,
    "rangeIndexVersion": 2,
    "invertedIndexColumns": [],
    "noDictionaryColumns": [],
    "bloomFilterColumns": [],
    "onHeapDictionaryColumns": [],
    "rangeIndexColumns": [],
    "sortedColumn": [],
    "varLengthDictionaryColumns": []
  },
  "quota": {},
  "query": {},
  "ingestionConfig": {
    "continueOnError": false,
    "rowTimeValueCheck": false,
    "segmentTimeValueCheck": true
  },
  "tierConfigs": [
    {
      "name": "hotTier",
      "segmentSelectorType": "time",
      "segmentAge": "1m",
      "storageType": "pinot_server",
      "serverTag": "DefaultTenant_OFFLINE"
    },
    {
      "name": "coldTier",
      "segmentSelectorType": "time",
      "segmentAge": "10m",
      "storageType": "pinot_server",
      "serverTag": "DefaultTenant_OFFLINE"
    }
  ]
}
Table_Schema: { "schemaName": "tiered", "enableColumnBasedNullHandling": true, "dimensionFieldSpecs": [ { "name": "product_name", "dataType": "STRING", "notNull": true } ], "metricFieldSpecs": [ { "name": "price", "dataType": "LONG", "notNull": false } ], "dateTimeFieldSpecs": [ { "name": "timestamp", "dataType": "TIMESTAMP", "format": "1MILLISECONDSEPOCH", "granularity": "1:MILLISECONDS" } ] }
m
In OSS Pinot, servers need to have locally mounted storage. You might be thinking of this feature which is not available in open source? https://startree.ai/resources/introducing-tiered-storage
k
Why tiered storage is not part of OSS Pinot ? Does StarTree ever pushes upstream enhancements, or is there a conflict of interests, where StarTree has to present advantages over OSS Pinot, so the OSS is actually "abandoned" on purpose ? I am really curious, as tiered storage is a core feature present in other systems like clickhouse and druid in their OSS distros.
m
Apache Pinot is an actively developing project with a wide set of contributors, and a ton of contributions coming from StarTree and various other companies. You can ask any StarTree specific questions in the StarTree community Slack I suppose.
l
What about if we mount an S3 as local files system using some like Mountpoint for Amazon S3, it has the following limitations: Mountpoint supports basic file system operations, and can read files up to 5 TB in size. It can list and read existing files, and it can create new ones. It cannot modify existing files or delete directories, and it does not support symbolic links or file locking , so not fullu POSIX would that be a problem? thanks
m
I know someone has tried NFS or something similar. As long as it can be read as a file I think it should work. Performance will be what you get
l
Just to add here in case some else finds the same the same issue, we tested 2 options: aws storage gateway for s3 or fsx lustre with S3 backend. Both serve a nfs that can be used local attached storage and configured as paths for cold storage.
m
That’s awesome @Luis P Fernandes, would you be able to share your learnings from that!
a
I'm working with Luis on this. FSx seems to work (still doing some stress tests), but cache needs to use SSD (trying to go the cheap way bet getting too many IO waits).
Some side effects on the experiences, having got issues with IO, cache storage full, etc., is that now we have many segments marked as BAD. We have 2 copies for the segments, and we have several scenarios: Scenario 1: segment ONLINE in server1 (s1) and ERROR in s2, segment files for s1 in cold with .tier file in default storage, segment files in s2 in default storage. Scenario 2: pinot says segments are in coldTier, both are in ERROR, and I have the segment files in coldTier and default storage. So, the questions are: • Having one copy ONLINE, how to force the segment to correct the OFFLINE copy? • Having both copies in ERROR but all the segment files, how to bring the segment to the correct state?
m
Deepstore has the golden copy. Calling reload api should fix the error segment. However, we need to understand why it is in error in the first place
a
Deepstore not configured... Yes, I know. We are in initial stages and not yet fully live.
m
Even so, there should be a copy on controller that servers can use to download. This is what you specify as controller dataDir
a
OK, I guess what I'm lacking is some understanding. I need to look again at the architecture. For this particular table, I have 1693 files in controller.datadir. Using segmentStatus API (http://localhost:9000/tables/'$tab'/segmentsStatus) or in GUI I have 1701 segments. Querying the table, I have partial results for 1587 segments. What segment status combination I get in query: Exists in controller and in servers is either in status GOOD or BAD but with one copy ONLINE Does not exist in controller and is GOOD in servers What segments are missing in the query: Exist in controller but on servers both copies are in ERROR. (1701-114=1587) So, for the query, only the existence of at least one good copy in servers makes a difference. On to correction. Most importantly, recover the segments that are missing in the query. Tried reload the segment (both using "Reload Segment" in GUI and via API /segments/{table}/{segment}/reload) and nothing changes. Since this is a realtime table, Force download is not applicable. How can I force pinot to get the copy from the controller?
m
1. Segments go to deepstore / controller.datadir after commit. This means that segments in CONSUMING state won't be available there. Can you check if this is contributing to the difference? I am guessing you have 8 partitions and hence 8 CONSUMING segments that are not yet committed. 2. Check external view in ZK browser. Segments should be either ONLINE or CONSUMING. If you see any other status, you should debug (look for segment name in server, call debug api etc) to understand why it is in ERROR state. 3. Query execution may prune segments based on query, partition, indexes. Are you saying that you got partial results in broker response? If so, check the status of those missing segments in external view. All in all if segments are in ERROR state, you need to debug why
a
Rookie mistake. Didn't realize that we are using Deepstore, because I was thinking wrongly about it. Tried debugging. Status in ZK for one of the missing segments in query: "elastic_kubeletstatsreceiver__0__17__20250710T1738Z": { "Server_pinot-server5_8098": "ERROR", "Server_pinot-server6_8098": "ERROR" In Zookeeper->PropertyStore->Segments { "id": "elastic_kubeletstatsreceiver__0__17__20250710T1738Z", "simpleFields": { "segment.crc": "2619840599", "segment.creation.time": "1752169102043", "segment.download.url": "http://pinot-controller2:9000/segments/elastic_kubeletstatsreceiver/elastic_kubeletstatsreceiver__0__17__20250710T1738Z", "segment.end.time": "1752173039613", "segment.flush.threshold.size": "1250000", "segment.index.version": "v3", "segment.realtime.endOffset": "1393549445", "segment.realtime.numReplicas": "2", "segment.realtime.startOffset": "1392299445", "segment.realtime.status": "DONE", "segment.size.in.bytes": "776020268", "segment.start.time": "1752168407274", "segment.tier": "coldTier", "segment.time.unit": "MILLISECONDS", "segment.total.docs": "1250000" }, "mapFields": {}, "listFields": {} } Used the following API call: curl -X 'GET' \ 'http://localhost:9000/debug/segments/elastic_kubeletstatsreceiver_REALTIME/elastic_kubeletstatsreceiver__0__17__20250710T1738Z' \ -H 'accept: application/json' Got: { "segmentName": "elastic_kubeletstatsreceiver__0__17__20250710T1738Z", "serverState": { "Server_pinot-server5_8098": { "idealState": "ONLINE", "externalView": "ERROR", "segmentSize": null, "consumerInfo": null, "errorInfo": null }, "Server_pinot-server6_8098": { "idealState": "ONLINE", "externalView": "ERROR", "segmentSize": null, "consumerInfo": null, "errorInfo": null } } } This does not provide me with much. I have the file in Deepstore: sh-5.2$ tar ztvf elastic_kubeletstatsreceiver__0__17__20250710T1738Z drwxr-xr-x pinot/pinot 0 2025-07-10 20:50 elastic_kubeletstatsreceiver__0__17__20250710T1738Z/ drwxr-xr-x pinot/pinot 0 2025-07-10 20:50 elastic_kubeletstatsreceiver__0__17__20250710T1738Z/v3/ -rw-r--r-- pinot/pinot 112964 2025-07-10 20:50 elastic_kubeletstatsreceiver__0__17__20250710T1738Z/v3/metadata.properties -rw-r--r-- pinot/pinot 775866612 2025-07-10 20:50 elastic_kubeletstatsreceiver__0__17__20250710T1738Z/v3/columns.psf -rw-r--r-- pinot/pinot 40676 2025-07-10 20:50 elastic_kubeletstatsreceiver__0__17__20250710T1738Z/v3/index_map -rw-r--r-- pinot/pinot 16 2025-07-10 20:50 elastic_kubeletstatsreceiver__0__17__20250710T1738Z/v3/creation.meta In server5: sh-5.2$ ls -lR elastic_kubeletstatsreceiver__0__17__20250710T1738Z* elastic_kubeletstatsreceiver__0__17__20250710T1738Z: total 0 drwxr-xr-x. 2 pinot pinot 90 Jul 31 19:22 v3 elastic_kubeletstatsreceiver__0__17__20250710T1738Z/v3: total 757840 -rw-r--r--. 1 pinot pinot 775866612 Jul 31 19:22 columns.psf -rw-r--r--. 1 pinot pinot 16 Jul 31 19:22 creation.meta -rw-r--r--. 1 pinot pinot 40676 Jul 31 19:22 index_map -rw-r--r--. 1 pinot pinot 112964 Jul 31 19:22 metadata.properties In server6: sh-5.2$ ls -lR elastic_kubeletstatsreceiver__0__17__20250710T1738Z* elastic_kubeletstatsreceiver__0__17__20250710T1738Z: total 0 drwxr-xr-x. 2 pinot pinot 90 Jul 31 13:59 v3 elastic_kubeletstatsreceiver__0__17__20250710T1738Z/v3: total 757840 -rw-r--r--. 1 pinot pinot 775866612 Jul 31 13:59 columns.psf -rw-r--r--. 1 pinot pinot 16 Jul 31 13:59 creation.meta -rw-r--r--. 1 pinot pinot 40676 Jul 31 13:59 index_map -rw-r--r--. 1 pinot pinot 112964 Jul 31 13:59 metadata.properties In coldTier storage, where ZK says is the segment: ll pinot-server6/elastic_kubeletstatsreceiver_REALTIME/elastic_kubeletstatsreceiver__0__17__20250710T1738Z/v3/ total 0 So, in summary for this segment: - ZK says it is in coldTier - coldTier has directory created but empty - both servers and controller have the segment files Where do I go from here?
In order to close this thread: there was an error in our cold storage configuration.
After correcting the configuration, we tried reloading or resetting the segments but to no avail: the missing copies of the segment remained. But after restarting the corresponding servers all missing/erroneous segments were downloaded and are fine now.
m
Thanks for closing the thread. What was the misconfiguration?
a
Just the normal replacing minus "-" for underscore "_": hard to see.
Regarding what brought us here, tiered storage. We ended up like this: - Local SSD for the default storage in each server - Common S3 storage for Deepstore mounted on both controllers (using Mountpoint for Amazon S3) - Cold tier in S3, using Amazon FSx for Lustre to transparently present S3 objects as files, with 1.2TB SSD storage for cache Performance is adequate, moving to cold tier is seamless and everything working without errors for the last 10 days.
❤️ 1
m
Glad to hear that