https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
This message was deleted.
# hamilton-help
s
This message was deleted.
👀 1
s
this path is called when checking the input to the DAG, so likely a function parameter argument
i.e. we’re double checking the input value passed in matches the type annotation of the function parameter
it appears that the “type”, i.e. what was annotated for the parameter, doesn’t work with the 
isinstance
function — so something we should handle.
do you mind sharing the function signatures of the initial functions you expect to be called?
a
The first function called is an instance of the _load_csv function. The signatures are
Copy code
pub_path: str, pub_mapping: dict[str, str]
s
What python version are you using? Sorry I'm AFK, but can you try changing dict[str, str] to just dict?
👍 1
a
i’m using python 3.9
This could be something related to the parameters i’m using. I’ll keep investigating and close the thread if i find the issue
s
@Elijah Ben Izzy ^
a
The issue was related to @extract_columns. I was using a dict as a parameter for it
e
Yeah so the problem is that it can’t do an isinstance cal with 
dict[str, str]
Ok — do you have the full stack trace? Usually we should be delegating to this function: https://github.com/stitchfix/hamilton/blob/64c014a3295fd5a8356dfa0d52292e302263c4bd/hamilton/base.py#L307
Ahh its this: https://github.com/stitchfix/hamilton/blob/cca5021a91ac4d20cbc7b825eff1c1b120ade51d/hamilton/experimental/h_spark.py#L88. Problem is we don’t delegate in the spark check, which we should. Getting out a PR and @Stefan Krawczyk and I will look. FWIW the quickest way to unblock you is to throw type-safety to the wind and make it a 
dict
🙂
OK, cool, have a PR out that I think should fix your issue. Let me know if the hack above doesn’t work — we’ll get this out with the change @Stefan Krawczyk is cooking up shortly!
s
@Anwar Brini just to clarify are you still blocked? Or?
Not sure I see the connection with extract_columns?
a
The problem seemed to be persistent yeah the extract_columns was not the issue. It was a separate issue
e
Any chance you tried declaring it as a dict, rather than a 
dict[str,str]
? The fix we’re getting out should allow you to do either, but for now the spark adapter breaks with generics.
Otherwise I can layer this on top of @Stefan Krawczyk ‘s change and give you a preview of our next release
a
I changed the signature but it doesn’t seem to be working.
e
:/ what’s the signature you’re using? Want to run my test against it so I can ensure we’ve fixed it.
a
Copy code
(drugs_path: str,
             drugs_mapping: dict[str, str]) -> ps.DataFrame:
e
Awesome — will take a look in just a bit. And you tested i with drugs_mapping as a 
dict
rather than a 
dict[str,str]
a
yes i did
👍 1
i was using @does decorator and it didin’t seem to work
so i removed it and changed the signature from dict[str,str] to dict and now it’s working
e
Ahh — so the signature fix (hack) worked but not with @does. Do you have a code snippet of how you were using @does?
a
Copy code
@does(_load_csv, csv_path='drugs_path', mapping_dict='drugs_mapping')
def df_drugs(drugs_path: str,
             drugs_mapping: dict) -> ps.DataFrame:
     pass
🙌 1
e
Will ensure this works. Do you have the signature of @does by any chance? I imagine it’s pretty straightforward
a
Copy code
def _load_csv(csv_path: str,
              mapping_dict: dict) -> ps.DataFrame:
👍 1
e
Will look in just a bit
Open in Slack
PreviousNext
Powered by