Hey quick question - is there a plan or a roadmap item in the coming weeks or months to migrate DAFT from Pyarrow + Pyiceberg to a more rust native implementation of the catalog and arrow-rs?

When you say migrate Daft from Pyarrow + PyIceberg -> Rust native implementation what do you mean? Are you referring to the iceberg catalog or the daft.Catalog or something else?

When I went through the source code, what I understood is that the iceberg reads/writes rely on pyarrow and not arrow-rs. And for catalog, daft uses pyiceberg for metadata operations. Would be nice if we shifted those to operations to rust as well. Might even see performance improvements. daft.Catalog implementation is fine. talking only about the IcebergCatalog and the I/O API for iceberg. My reference point - https://github.com/apache/iceberg-rust

@Colin Ho would probably have more thoughts on this.

Hey @Navneeth Krishnan! Yeah that would be nice, but it's currently not in scope for us in the next few months. You're right that it could be great for performance, is there any other specific features / improvements in iceberg-rust you had in mind that would be beneficial for us?

I think iceberg-rust overall is quite ahead in terms of iceberg V3 spec adoption compared to the python version. I think that is also worth looking into. Iceberg v3 spec is kind of a game changer for CDC workloads and if we can have Daft also start support for v3 spec would be great.