This message was deleted.

This one? https://hydra.cc/docs/intro/. I wonder if other users has the same challenge.

I've heard about hydra but never used it. I think we need a bit more info here (hope other people can weigh in). Cause I'm unsure if there's potential for ploomber to improve env.yaml management or it's better to have an integration with Hydra

By unwieldy I mean when using complex configs I see I could either: • pass every parameter in

params

in pipeline.yaml • set config as dict and pass it in params, but this makes function signature not reflect what is actually needed

this is great feedback. thanks! I agree with you, when you have tons of parameters passing them one by one is tedious. perhaps something that automatically grabs them would work? this reminds me of pytest fixtures, where you declare function arguments and then pytest passes the appropriate values. for example, if your env has:

key: value
another: another-value
final: last-value

and your task:

def some_task(key, final):
    passs

ploomber could inspect the function's signature and pass key and final (but not

another

). thoughts? the tradeoff is that the pipeline.yaml is not explicit anymore, since we'd have to dynamically pass the params. it'd look something like this:

task:
   - source: functions.some_task
     params: auto # special value to have ploomber inspect the function signature

Hmm that might be useful, but I meant something else - the problem when we have tons of parameters env.yaml gets cluttered and it's harder to manage like hydra configs which can use single file for each config. I actually tried using configs from other files but this has the problem that ploomber does not see when a config gets updated.

I actually tried using configs from other files but this has the problem that ploomber does not see when a config gets updated.

you mean using configs for third party libraries? like you use hydra, and change the config but ploomber won't detect the changes?

Other than that, it might be useful to have something like automatic serialization

we have serialization of products. is this what you're looking for? https://docs.ploomber.io/en/latest/user-guide/serialization.html

Wow, that's not what I meant, but it's really great to know this 😄

aaah. i got it now. are you passing configs from other projects because they're not written with ploomber? you can use

resources_

for this: you can pass a path to a file and ploomber will look for changes there to re-run a task. the path is also accessible to the task so you can use it for loading the config https://docs.ploomber.io/en/latest/api/spec.html#tasks-params-resources