Is it possible to add S3 deep storage after a cluster has ingested data? The documentation I found uses a s3 url for the controller.data.dir property which in my poc currently points to a directory on the fs on the controller.
k
Kishore G
08/01/2021, 9:44 PM
It is possible theoretically but it’s not automatic.. the url of the segments are stored in segment metadata in Zookeeper
So you can change them to S3 and it will work seamlessly
b
Bruce Ritchie
08/01/2021, 9:45 PM
I'd assume you would have to copy over the data on the fs? never mind, doesn't matter. No way I'm going to go to the trouble of rewriting zookeeper data.
k
Kishore G
08/01/2021, 10:08 PM
It’s simper to delete and upload the same segment again
b
Bruce Ritchie
08/01/2021, 10:16 PM
Indeed. Definitely something to script.
k
Kishore G
08/01/2021, 10:25 PM
Feel free to contribute the script if you can generalize it