Hey Team, I’m working on a custom connector to be able to create daft data frames from spark data frames and vice versa. Wanted to get some guidance on it. I don’t want to load data from spark partitions into a driver node and I don’t want to load data from ray worker nodes into head node. I want it to be scalable. Are there any limitation or best practices or guidance around this? Thank you!

Hey Nish, daft and spark are two different systems so any data transfer is going to be very expensive. Have you considered writing the data out and reading it back in?

Hey @jay Yes but the thing I’m trying to overcome is with daft on databricks with unity catalog I can only read from external tables and not from managed tables. Daft also cannot read from tables with deletion vectors. I am creating a library using daft but the users of this library use spark to read and write data and I want to have a way to convert this spark dataframe into daft dataframe in a scalable fashion. Using spark for io and daft for processing unblocks a lot of overhead. Adding a hop with writing the data out into a file or an external table introduces latency and additional clean up. I want to have something that’s 1:1 translation between the two

would performance be decent for ray -> daft data frames?

Ray Data is better because daft can re-use the same Ray objects in object store, avoiding the need to transfer data between two systems In the Spark case, data needs to move between Spark and Ray which is very expensive