Hey, I know this isn't a purely Hamilton question, but I thought you might have good insights for me as this seems like a common expectation of Hamilton pipelines. If you remember, we were hoping to create a feature engineering platform to allow data scientists to write normal pandas code (with patterns such as groupbys and joins, etc) and run it at scale. After struggling with Dask a bit, I have found the following - and I wanted to run it by you guys to check if I'm missing something: • The idea of writing normal pandas code and having it be scalable is unfeasible. Dask for example provides a deceivingly simple interface, but if you don't understand the underlying mechanism to write proper parallel code, you will have a fragile memory-hungry pipeline. (here's a link to a blogpost around their experience). Now this isn't to say Dask is not the right tool, it is highlighting that we need all the people who write feature transformations to have an understanding of Dask and parallel computing. This poses a great issue as this might be unreasonable learning overhead that we were hoping to avoid. ◦ An example of this learning overhead is that it seemingly is not a good idea to just call

groupby

. We should instead use

map_partitions

which is a parallel compute concept. Or another example I've found is that

groupby

's that aggregate using custom functions are harder to scale and less optimized (can't find the source right now).

how old is that post?

The article is from 2021, but also wouldn't modin have similar pitfalls specially at scale?

So technically both should be scalable — they’re both to enable distributed compute over larger than memory datasets. Modin just takes a different approach to it than Dask; Modin can use dask as backend IIRC. If you’re really worried about scale then pyspark should be a consideration; it does support pandas on pyspark if you want people to continue to use pandas (I can point to an example repo using Hamilton).

Makes sense. It seems like we'll just have to choose a scalability tool and go through the learning overhead. Thanks Stefan!