So on this polars lazyframe stuff @Stefan Krawczyk I've basically added another polars plugin to the source which uses lazyframe instead of dataframe. First up a) is that the correct path for whats missing in Hamilton, it seems to work here now I can toss lazyframes around and then run collect at the end and have it materialize them b) if I can get the other readers and writers squared away do you want it as a PR into the codebase?

Exciting! Seems like its different enough and would probably be a good addition to the codebase. Some pointers: 1. h_polars — this has anything that gets directly imported. You’ll likely want to add a result builder (if you’re not already) that handles materialization, or leaves it as an option 2. polars_extensions — this gets automatically included. You might want to be clever about the mode (lazy or not) and how we select the constructs. Hard to say without looking at your code. But yes! love contributions, and polars is absolutely on our roadmap.

Hey Elijah, I saw your commits on the Dataframe flavor, I've done most of what you mention above. I did wonder about a combined polars extension version but then there's stuff like DATAFRAME_TYPE and I've not tested it but I don't think its going to take a Union type hint in there is it to allow an either/or scenario? Similar with columns not being series and all that fun stuff. There will be a bunch of shared code that could get moved out to a combined block of logic so that the hard work is already one.

Awesome! Maybe best is to point us to some code or make a PR (if it feels at that stage) to ground the conversation?

yeah no worries. Let me get it squared away and I'm dump up a PR and we can go from there.

Great, looking forward to it

Yeah we assume one data frame type per extension. So it'll need to be a separate extension.

Okay so I've chucked https://github.com/DAGWorks-Inc/hamilton/pull/775/files into the repo, this currently just fixed the CSV reader and writer for both lazyframes and dataframes in polars, but I was looking for feedback on the layout before I did more. So far I created a polars_shared file with a ReaderWriter class thats pretty simple. I can't really push the kwargs down because they differ for dataframes and lazyframes. Well I could but there's been a big if else block that doesn't seem to make sense at least. Anyway it'd be good to know folks thoughts so I can get this done properly and not running off a temporary fork 🙂

@Tom Barber sounds good. Will look today/tomorrow 🙂

Thanks for the feedback Stefan, I stuck another couple of questions in there to try and get it right

Thanks @Tom Barber . I responded. I’m happy to jump on a call if that’s faster to clarify what I mean. Would you like to do that?