Hi there, I was planning on working on this issue of adding a custom delimiter to the

write_csv

option https://github.com/Eventual-Inc/Daft/issues/3786 but I realized that since I'm using one field of PyArrow's csv WriteOptions: https://arrow.apache.org/docs/python/generated/pyarrow.csv.WriteOptions.html#pyarrow.csv.WriteOptions Might as well implement support all the fields, as it shouldn't be too difficult. This is the original PyArrow class:

@dataclass(kw_only=True)
class WriteOptions(lib._Weakrefable):
    include_header: bool = field(default=True, kw_only=False)
    batch_size: int = 1024
    delimiter: str = ","
    quoting_style: Literal["needed", "all_valid", "none"] = "needed"

    def validate(self) -> None: ...

I want to implement the corresponding class in Rust like this:

#[derive(Debug, Clone)]
pub struct CSVWriteOptions {
    pub include_header: bool,
    pub batch_size: usize,
    pub delimiter: char,
    pub quoting_style: QuotingStyle,
}

#[derive(Debug, Clone)]
pub enum CSVQuotingStyle {
    Needed,
    AllValid,
    None,
}

However there's a couple of obstactles, the first one it seems that Rust doesn't support default fields, so I guess I create a special constructor for default I guess? But the main problem is, the

quoting_style

field, it accepts only specific kinds of strings, normally this is fine, but the problem is that when that Rust struct gets passed to Daft's Python CSV Writer class which uses those options with PyArrow, it accepts a string for

quoting_style

, not an enum, what is the cleanest approach to this? As far as I'm aware you can't have specific values for strings in Rust, you have to use an enum. Should I just create two structs, the one above, and a new one that's the same but accepts

quoting_style

as a String and the original

CSVWriteOptions

has a function that converts it a new struct that turns the

quote_style

into a string? How does Daft deal with Python's string literal and Rust enum interoperability? I feel like this is something you have encountered before and I want to know your standard of doing this.

I would actually prefer if we didn't support all pyarrow csv options. arrow_csv doesn't support all of these options it seems and it would be good to keep the door open so that we can switch over

oh alright then, I'll just implement the delimiter option, thanks!

We currently use python but we'd like to move to our rust writer and rip out pyarrow