https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Some thoughts and issues I've run into when doing ...
# hamilton-help
m
Some thoughts and issues I've run into when doing a Spike on Hamilton. This is an internal Python data analytics project that has expanded beyond sanity. I'm exploring Hamilton as a possible framework to guide data scientists where to put code, remove state, and to untangle dependencies. At a glance it seems like a perfect fit but I've run into some issues but maybe I'm thinking about it the wrong way. I have a few items but let's start with this: How do I dynamically parametrize a node? For example I have a function in the current code that takes a parameter, lets say X, Y and Z. Other nodes depend on the output of that function depending on config and other node outputs. Sometimes all three outputs are needed. I broke them up into example_output_x, example_output_y, and example_output_x. using the parameterize decorator. Not pretty but let's finish the swing. However, this "node dimensionality" expands since there is actually another parameter, let's say an int so now we have example_output_x_1, example_output_x_2, ... example_output_z_3. At this point I paused because I can't propose that. I'm trying to simplify, not make the project and graph more complex. Please don't say Pandas since the whole project is full of Pandas with thousands of dynamic columns 🙂. If you say Pandas or some dimensional datatype, please tell me why there is no other way. What am I missing? I'm coming from mostly a graphics pipeline C++ with a heavy slant on pure functional stateless designs.
s
@Mattias Fornander US want to jump on a quick call?
what I’m hearing is - base case:
Copy code
# omitting types for brevity
def my_func(x, y, z):
   # some logic
   return output_x_y_z

# downstream
def some_other_func(output_x_y_z):
   # depends on the particular parameterization of x, y, z.
   return ...
but you want to parameterize even more:
Copy code
# omitting types for brevity
def my_func(x, y, z, int_value):
   # some logic
   return output_x_y_z_int_value

# downstream
def some_other_func(output_x_y_z_int_value):
   # depends on the particular parameterization of x, y, z + int value
   return ...
Is that right? And the challenge you’re running into is, (a) naming to handle what you call each thing, (b) handling logic downstream of the parameterization, e.g. what does it depend on? To help me, could you sketch a diagram of how it should work, or provide some pseudo code (maybe even code what you’re trying to move away from?)? also how will things be executed, will someone want to create a dataframe with all those parameterizations? or?
In case it’s helpful here’s one

overview of reuse & parameterization▾

(notebook). With more info I can direct which of the constructs could be a better fit here; the video explains a few dimensions to think about…
m
Thanks for replies! Stuck in meetings today but let me dig in again tomorrow.
👍 1
3 Views
Open in Slack
PreviousNext
Powered by