Is it possible to add S3 deep storage after a cluster has ingested data? The documentation I found uses a s3 url for the controller.data.dir property which in my poc currently points to a directory on the fs on the controller.

It is possible theoretically but it’s not automatic.. the url of the segments are stored in segment metadata in Zookeeper

I'd assume you would have to copy over the data on the fs? never mind, doesn't matter. No way I'm going to go to the trouble of rewriting zookeeper data.

It’s simper to delete and upload the same segment again

Indeed. Definitely something to script.

Feel free to contribute the script if you can generalize it