Now iceberg is standardizing their FileIO API. Would it make sense for StarRocks to use that and start supporting file formats other than parquet? I'm hoping future formats like Lance and vortex looks promising for random access data and also index management

@Kesav Kolla thanks for noticing. it totally makes sense to support those file format. we can support by using iceberg fileio api, but to push performance to the extreme, we have to read lance/vortex in C++ which requires some efforts. do you mind sharing your case that why lance/vorttex is so important in your scenario?

Both file formats offer better/customized encodings which will reduce the cost of storage. Both formats offer ability to decode without doing entire file decompression. So the CPU cycles and probably network IO involved to read metadata is very minimal. They seem to offer better random access efficiency. Lance is promising with the addition of indices support.