Is it possible to add S3 deep storage after a cluster has ingested data? The documentation I found uses a s3 url for the controller.data.dir property which in my poc currently points to a directory on the fs on the controller.
08/01/2021, 9:44 PM
It is possible theoretically but it’s not automatic.. the url of the segments are stored in segment metadata in Zookeeper
So you can change them to S3 and it will work seamlessly
08/01/2021, 9:45 PM
I'd assume you would have to copy over the data on the fs? never mind, doesn't matter. No way I'm going to go to the trouble of rewriting zookeeper data.
08/01/2021, 10:08 PM
It’s simper to delete and upload the same segment again
08/01/2021, 10:16 PM
Indeed. Definitely something to script.
08/01/2021, 10:25 PM
Feel free to contribute the script if you can generalize it