Hamilton Open Source

Hey! I thought we had some tests for this — you may have found a bug! — mind pasting a little code/the error to help us get started debugging?

yep, seems like we don’t have a test using pyspark’s pandas dataframe type.

I think I recreated the issue; will be able to give an ETA to the fix tomorrow.

<@U04JZKAM308> actually do you mind pasting the code the extract_columns decorator operates on top of?

Or some pseudo code .. don't want to claim fixing without verifying your situation :)

<@U04JZKAM308> could you try this install of Hamilton? I believe I have fix for your issue. Would you mind confirming that I have fixed it by installing directly from my branch?
That is do:
```pip install <git+ssh://git@github.com/stitchfix/hamilton.git@add_extensions>```
in the python environment you’re prototyping Hamilton with, and then trying your code.

Otherwise in terms of scaling, I would also consider prototyping with <https://modin.readthedocs.io/en/stable/|Modin> — they implement more of the pandas API than pyspark has.

Thanks Stefan! It’s working with the fix

Ok cool — I’ll do some testing and prep a release for early next week.