https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
This message was deleted.
# hamilton-help
s
This message was deleted.
m
I suppose 
check_output_custom
 should work, right, but wanted to check if I'm maybe missing something easier
I cannot use pandera for some annoying version resolution reason within my project 😕
👍 1
e
Yep, 
check_output_custom
is what you want — a bit cumbersome, but not crazy. Happy to accept ideas on how to do this better. We could think about adding stuff. Also I’m a bit of an expert on version management in python (through no fault of my own), so if you think its solvable…
m
thanks, but I don't think it's solvable... the issue is that I'm using Feast and I'm upgrading to the newest version, and they sometimes just require lower versions 😕
they used to require typeguard 3 but they lowered it to 2
they have a tendency to do that. e.g., I see numpy is downgraded (though not as drastically) as well
not sure if anything can be done easily
e
Hmm — not seeing here? Transitive dependency? https://github.com/feast-dev/feast/blob/master/pyproject.toml#L2
m
e
Oh that’s really sloppy on their part
Makes sense if its an application but as an SDK, yikes
m
if I remember from past, I think there was some issue on 3 so they just hardcoded a version
might need to open some tickets there, get some answers maybe
e
facepalm
Maybe there’s an older version of pandera?
m
no, none that works unfortunately
it's not a huge deal, I'll deal with it, but extremely annoying
e
Yeah that’s rough
Also you can sometimes just clobber dependencies if you’re OK living fast/loose
m
this is not worth it
I use very simple checks, I'd prefer to just use a custom validator
e
Fair — so yeah, custom validation is pretty straightforward (you just pass in a class)
If you want to make it more ergonomic you can define your own decorator with your API that just calls out to that
m
nah, again -- not worth it 🙂
e
This is btw why we’re obsessive about keeping the hamilton dep footprint small
👍 1
👍 fair
m
I think tbh your solution with pandera is what it's supposed to be
I'm hoping I can go back to using it at some point, but until then I want to make my workaround as simple as possible
e
Yep and usually the code to do this stuff is quite simple
m
yeah, anyway, thanks for your response
e
Yep! Good luck
👍 1
s
@Elijah Ben Izzy how hard would it be to provide a validator that registers itself like the default validators? or is it simpler to go the 
check_output_custom
route?
e
Should be simple but its not something we’ve designed for yet — you’d have to change this list: https://github.com/DAGWorks-Inc/hamilton/blob/b207db71a79c12413b75f981277d813a82f1c89d/hamilton/data_quality/default_validators.py#L399
We should unify this with the “registry” concept we have for data i/o
Open in Slack
PreviousNext
Powered by