https://pinot.apache.org/ logo
#getting-started
Title
# getting-started
b

Bruce Ritchie

08/01/2021, 6:31 PM
Is it possible to add S3 deep storage after a cluster has ingested data? The documentation I found uses a s3 url for the controller.data.dir property which in my poc currently points to a directory on the fs on the controller.
k

Kishore G

08/01/2021, 9:44 PM
It is possible theoretically but it’s not automatic.. the url of the segments are stored in segment metadata in Zookeeper
So you can change them to S3 and it will work seamlessly
b

Bruce Ritchie

08/01/2021, 9:45 PM
I'd assume you would have to copy over the data on the fs? never mind, doesn't matter. No way I'm going to go to the trouble of rewriting zookeeper data.
k

Kishore G

08/01/2021, 10:08 PM
It’s simper to delete and upload the same segment again
b

Bruce Ritchie

08/01/2021, 10:16 PM
Indeed. Definitely something to script.
k

Kishore G

08/01/2021, 10:25 PM
Feel free to contribute the script if you can generalize it