It seems the s3 deep store controller dir cannot end with / I had following config

controller.data.dir=<s3://pinot-data/pinot/controller_data/>

When controller tried uploading completed realtime segment. The path had // , between controller data dir and the table dir which lead to failure to copy and then deleting the temp segments

2022/01/18 12:08:31.989 INFO [S3PinotFS] [grizzly-http-server-5] mkdir <s3://pinot-data/pinot/controller_data/test>
2022/01/18 12:08:32.008 INFO [S3PinotFS] [grizzly-http-server-5] Copying uri <s3://pinot-data/pinot/controller_data//test/test__3__0__20220118T1147Z.tmp.c88bn119-kdei-kf87-ki87-z937482d8x5> to uri <s3://pinot-data/pinot/controller_data//test/test__3__0__20220118T1147Z>
2022/01/18 12:08:32.047 INFO [S3PinotFS] [grizzly-http-server-5] Deleting uri <s3://pinot-data/pinot/controller_data//test/test__3__0__20220118T1147Z> force true
2022/01/18 12:08:32.088 ERROR [SegmentCompletionFSM_test__3__0__20220118T1147Z] [grizzly-http-server-5] Caught exception while committing segment file for segment: test__3__0__20220118T1147Z
java.io.IOException: software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist.

Removing / at the end of controller dir resulted in correct path

controller.data.dir=<s3://pinot-data/pinot/controller_data>

Is it some know issue in 0.9.3 ?

did it used to work in earlier versions with the trailing slash?

didn't tried in earlier versions. I was following the tutorial which had trailing slash in controller.data.dir https://docs.pinot.apache.org/basics/data-import/pinot-file-system/amazon-s3#controller-config

ah ok. I remember that @Dunith Dhanushka noticed some weirdness with a trailing slash, but for him I think it was removing the slash rather than having two slashes! 🤔