S3PinotFS.java

adds support for S3 for Pinot, however

listFiles

is inconsistent with the rest of implementations. It skips directories, which, in turn, breaks retention mechanism for deleted segments (as it locates "garbage" tables by scanning directories in the configured folder):

@Override
  public String[] listFiles(URI fileUri, boolean recursive)
      throws IOException {
    ImmutableList.Builder<String> builder = ImmutableList.builder();
    visitFiles(fileUri, recursive, s3Object -> {

      // VVVVVVVVV
      // TODO: Looks like S3PinotFS filters out directories, inconsistent with the other implementations.
      // ^^^^^^^^^

      // Only add files and not directories
      if (!s3Object.key().equals(fileUri.getPath()) && !s3Object.key().endsWith(DELIMITER)) {
        builder.add(S3_SCHEME + fileUri.getHost() + DELIMITER + getNormalizedFileKey(s3Object));
      }
    });
    String[] listedFiles = builder.build().toArray(new String[0]);
    <http://LOGGER.info|LOGGER.info>("Listed {} files from URI: {}, is recursive: {}", listedFiles.length, fileUri, recursive);
    return listedFiles;
  }

To workaround this one may set segment retention to 0d and thus avoid collecting garbage. Has this been done intentionally or is it a bug?

This seems to be very old change of mine. I think this flow might have got introduced later on. Will check and fix