No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

Also it's unclear to me how do I get parameters for a given task - this does not show when running `ploomber status`

Anyway, the grid example on using MLFlow was great, I came looking for running param grids but the tutorial matched what I needed better than I expected!

currently, there's no way to change the default name "task_name_{i}" but we have an open issue about it. what you can do; however, is to customize the output using `[[param_name]]`, see the example here: <https://docs.ploomber.io/en/latest/api/spec.html#id7>

regarding the status, you can run `ploomber interact`; then do `dag[task_name]` params and that will return the parameters for that task

is there a different way to parametrize products in tasks with grids?

I have a function that makes a plot, and if it doesn't see .png extension it adds one, so ploomber fails because he doesnt see 'plot_product-0' ('plot_product-0.png') was generated

what happens if you put `product: plot_product.png` ? in such case, ploomber will generate `plot_product-{number}.png` . would that still break the plotting function?

I think it should work because we parse the “{{value}}” with <https://docs.python.org/3/library/ast.html|ast.literal_eval> which supports dictionaries. Can you try and let me know if it works? 

e.g., `models` would be a list of dicts:

```models:
    - p1: a
      p2: b
  
    - p1: c
      p2: d```
this isn't ploomber-specific. Just google around for python and yaml config

Is there a way of combining parameters though?

like with {**some_params, **additional_params} in Python?

Not currently. We have an open issue. We'll allow to define a function using Python and have the returned value as parameters.

Update: it seems to be a problem in powershell, command line works without problems

nice, so did you get it to work? thanks for giving ploomber a try, don't hesitate to send more questions to this channel!

yes, managed to get it working in CLI, thanks!

Sorry for the trouble! Another user reported a similar problem, I also asked him to disable the extension but it didn't work, so I'm investigating the issue and will push a fix asap.

Can you provide some extra details for me to debug this:

• how large is your pipeline? (in terms of number of tasks)
• which version are you running (ploomber and jupyterlab)

as a temporary fix, you  may uninstall ploomber from the process that runs JupyterLab, then you can install it in a separate environment to use the CLI

Hey Eduardo thanks for your time and all good. its not big yet, I have like a folder of pipelines. I just found a workaround to rename my pipeline.yaml to something else and that seemed to stop it ( or it could've been because I reset the prompt and conda etc.

I just did a reset so all pretty new I think

```Selected Jupyter core packages...
IPython          : 7.33.0
ipykernel        : 6.15.0
ipywidgets       : 7.7.1
jupyter_client   : 7.3.4
jupyter_core     : 4.10.0
jupyter_server   : 1.18.1
jupyterlab       : 3.4.3
nbclient         : 0.5.13
nbconvert        : 6.5.0
nbformat         : 5.4.0
notebook         : 6.4.12
qtconsole        : not installed
traitlets        : 5.3.0```


alright. keep an eye on <https://github.com/ploomber/ploomber/issues/894> to know when this is fixed. thanks for your feedback!

yes, renaming also works. since the plugin wont find any pipelines and stop parsing it

hey <@U038HCFR6MS>, I did a quick test to see if we can improve the performance when loading the pipeline, can you try it and see if it improves things?

```pip uninstall ploomber
pip install ploomber git+<https://github.com/ploomber/ploomber@dag-reload>```
after testing you'll probably want to get back to a stable version:
```pip uninstall ploomber
pip install ploomber==0.19.7```

Hi! I’ve parameterized a notebook so that I can use two different upstream datasets for the same analysis. Is there a way I can inject the parameters for a specific upstream or a specific target?

I see two options here, one is to put `analyze_data.ipynb`  twice in your `pipeline.yaml`(but with different names, example `name: analyze_data_a` and `name: analyze_data_b`)  and put different parameters to each task. then, you can execute selectively with `ploomber build --partially analyze_data_a`

the second option is to have a single instance of `analyze_data.ipynb` , you can use an `env.yaml` to <https://docs.ploomber.io/en/latest/user-guide/parametrized.html|parametrize the pipeline> and then you can switch the parameters at runtime: `ploomber build --env-parameter some-value` . Ploomber automatically exposes all the values in the `env.yaml`  in the CLI, you can see them if you do `ploomber build --help`

Do any of these options solve the issue?

The first one is what I have working right now, but partial building helps some. I like writing code interactively (for better or for worse). Will the partial build leave the filled/injected parameters in place so that I can go through and run all the cells/inspect the results myself?

(i can also just run off and test that it does lol)

&gt;  Will the partial build leave the filled/injected parameters in place so that I can go through and run all the cells/inspect the results myself?
are you referring to the execute notebook (the one that appears in the product section)? If so, yes. the output notebook will contain the injected parameters

What about the input/source notebook? I work mooostly with R/.rmds so it’d be nice if the injected parameters persist there

Ah, in that case. you can manually inject the parameters to the source with `ploomber nb --inject` , but you suggested that before. so maybe im not following (?)

image.png

right! heres a snippit of a pipeline I’m working on…
so `compare-adultsVsChildren` and `compare-semcor` both use the same underlying R script `compare_models.R` (with a `name:` argument in the yaml). I’ve parameterized them so that they take in either `adultsVsChildren`  or `semcor` and each target has different upstream tasks (for their different datasets). But right now, when I `ploomber nb --inject`  I only see the parameters/upstream paths for `compare-semcor` . I’d like to be able to ask ploomber to inject the parameters for `compare-adultsVsChildren`  instead if I want to interact with that dataset in R instead… any ideas?

these days I’ve been going in and commenting out the other targets that use that script so that it only injects the parameters for the target I want to interact with, but that feels pretty messy

ah. got it! so the problem is that `compare_models.R` appears once in the pipeline. so only the first parameter is injected. We don't have a way to easily achieve what you want, unfortunately, but I think we should. Let me open a PR to describe your problem and get your feedback. it won't be difficult to implement :slightly_smiling_face:

Alright! Yeah, I’d love to be able to inject the parameters for a specific target I think that’d get to the root of it for me. Thanks!

please share your feedback <https://github.com/ploomber/ploomber/issues/902>

Yes, you can use two env files (e.g. `env.yaml` and `env.prod.yaml`), then you can switch them at runtime. More in <https://docs.ploomber.io/en/latest/user-guide/configuration.html|the docs>. - does this solve the issue?

something like this where you can just pass an array to parameterize over could be nice but multiple env files will do the job

<https://docs.pytest.org/en/6.2.x/parametrize.html>

you might be looking for the <https://docs.ploomber.io/en/latest/cookbook/grid.html|grid> feature then

can I parameterize outputs based on the grid

yes, we <https://docs.ploomber.io/en/latest/api/spec.html#tasks-grid|added that> in version 0.17.2 so make sure you have that or higher. you need to use `[[` `]]` placeholders

```tasks:
  - source: random-forest.py
    name: random-forest-
    product: 'n_estimators=[[n_estimators]]/criterion=[[criterion]].html'
    grid:
        n_estimators: [5, 10, 20]
        criterion: [gini, entropy]```

is it possible to pass the db client to a py file?

I can import it manually but was wondering if ploomber could help with that

Not at the moment. We got a similar request so we'll work on it soon. But right now you need to import it manually 

trying to create a task dependent on a grid

ah. this looks like a bug. let me try to reproduce it

before you do anything let me dig in a little

I had `test-grid-`  and then created another task `test-grid-downstream`

the issue is `test-grid-downstream` implicitly depends on itself

ah yes. that will fail since it cannot resolve as a DAG

I'm curious though. There isn't a way to use the same grid across multiple tasks

also can I use the grid parameters in the name?

re multiple tasks: is this what you're looking for? <https://github.com/ploomber/ploomber/issues/602>

could I make the names `test_grid-[[param]]`

&gt;  could I make the names `test_grid-[[param]]`
there's also an open issue about this :sweat_smile:

In regards to your comment the parameters should be passed through to each source

keep an eye on this: <https://github.com/ploomber/ploomber/issues/698>

we'll probably close it in the next sprint cycle

since parameters could subtly change the output and downstream tasks may have behavior dependent on parameter values

i.e. if I pass `train`, `eval` to the grid is the task-0 always associated with `train`

hmm I guess thats a loaded question since the grid could have multiple variables

interesting question. i never thought about it. i took a quick glance at <https://github.com/ploomber/ploomber/blob/5ae9ebf3de66612b5f8a0a6fb3d14262b4a64f2b/src/ploomber/util/param_grid.py#L54|the source code>. I think it should be since we generate the grid by iterating over the dictionary of parameters and beginning in Pytyhon 3. 6 (I think) dicts iterate in order

but if you find that they're not, let me know

yea all good. at least the name in the outputs helps

so I can manually disambiguate and do the separate pipeline behavior in application code

please keep the feedback coming, it allows us to prioritize what to work on. otherwise we have to guess what's useful for our community

Hey guys, did anyone come to ploomber after using Metaflow? What are your thoughts? It seems like metaflow cannot handle probably most important ploombers functionality, that is tracking execution of DAG steps. Am I right?

Hey folks having trouble retrieving the API key? Can someone help me out? Thanks!

I'm having the same issue as <@U03PR0DN15E>

<@U03PR0DN15E> <@U03P181L7L2> are you in the 106 room? If so, raise your hand so we help you 

We're in room 106 running the workshop. Wanna come here?

Sure thing, we end here around 12, the sessions are also recorded so you can view it later. Feel free to ping us in the Scipy slack channel

Alright, there was something off with the registration flow
I fixed it, creating a new user will work fine now!

Worked for me too! thanks :slightly_smiling_face:

Feel free to share more feedback, it helps us improve the product and make it better!

Currently, in ploomber, after running ploomber nb remove, the parameters cell is empty. This causes checkers to complain as product variable doesn’t exist.

I would be great if instead of being empty, it was a TypedDict.

anyone here using <https://quarto.org/docs/get-started/|Quarto>?

This is a great point! We’ll try doing it on our next posts, we’ve automated this via buffer and I see <https://support.buffer.com/hc/en-us/articles/1500005374202-How-alt-text-works-when-scheduling-Twitter-posts-through-Buffer|that’s possible>. Any post in particular you want to retweet now so we’ll add it manually?
<@U0164GZJ5EK>

thanks for the heads up <@U039FB8GYRL> - I'll ensure we add alt text for upcoming posts :slightly_smiling_face:

<@U02KLC3P7DY> we just committed a fix for this issue - <https://github.com/ploomber/ploomber/issues/878#event-7041763576|https://github.com/ploomber/ploomber/issues/878#event-7041763576> it'll be available on our next release

hi all, has anyone had merge conflicts when storing product ipynb files in git repos? A user <https://github.com/ploomber/ploomber/issues/930|opened this issue>, and we provided a manual fix. But we're wondering if we should implement it as a feature. maybe:

```tasks:
  - source: nb.ipynb
    product: out.ipynb
    strip_product_metadat: True```

the 5th cell in <https://github.com/vedanthv/ml-pipeline-config/blob/main/diabetes_ml/tasks/train_model.ipynb|train_model.ipynb> is overriding the injected values. remove it and it'll work. I think what happened is that you used `ploomber scaffold`, then set `extract_upstream` to false and then somehow the parameters cell ended up being an empty cell, and the one that ploomber adds by default (the one with `upstream =None`) ended up below it. but please share the steps you followed so we can capture this error and alert future users :slightly_smiling_face: thanks for your feedback!

thanks for your help <@U0164GZJ5EK>. I removed the `upstream = None` cell that was overriding the injected values

But I'm not able to get PyCaret working in the same notebook even though I have imported the required modules whereas if I run the pycaret code standalone in another notebook it works well

I get a `Name Error : fold_shuffle is not defined` error

i looked at your notebook, you are calling a setup function (im guessing this is from pycaret) and passing a fold_shuffle variable, but the variable isn't defined anywhere

ensure you declare `fold_shuffle`  before calling setup

yes that was indeed the issue. I forgot to declare `fold_shuffle`  and `fold` parameters in my yaml file! Now everything works including the ML Flow logging :slightly_smiling_face:

I love how Ploomber integrates with mlflow. Definitely going to use Ploomber for my projects from now on. Thanks for your help!

Thanks for using Ploomber! Don't hesitate to ask any other questions!

Can I share your project on our socials? Also if you have a Twitter account, I can tag you 

yes definitely, There is a bigger project I want to implement ploomber to, it will take me a couple of days to set it up on my project, I'll let you know once that's done!

Untitled.txt

If I `⌃c` after it freezes, I get the following traceback

<@U0164GZJ5EK> tell him about our launched engine
<@U0327AHTCUU> great timing!

Check <https://github.com/ploomber/ploomber-engine|this one out>
Feel free to star it :star-struck:

so ploomber does not restrict cpu/memory, so there isn't any control to bump it up - ploomber adds some cpu/memory overhead but it's minimal. I'd start by <https://github.com/ploomber/ploomber-engine#notebook-profiling-cpu-gpu-ram|profiling> the notebook with papermill (we will integrate this into ploomber we haven't finished yet)

you can also try running the notebook in isolation with `ploomber task`

one of the problems with papermill is that it runs the notebook in a separate process and it has to communicate to the kernel, so it might happen that the kernel freezes and papermill enters in a weird state - we're working in a full replacement for papermill to fix those issues so stay tuned (actually, the profiling engine is an early version of what the new ploomber engine will be so give it a try)

Interesting, so it seems to work using `ploomber task`. For my own edification, why is that?

&gt; so it might happen that the kernel freezes and papermill enters in a weird state
this sounds likely.

Side note: you may want to add `memory-profiler` as a dep of `ploomber-engine` (I had to `pip install` it separately).

Running `mprof run papermill /path/to/notebook.ipynb &amp;&amp; mprof plot`  works as expected:

are you using the parallel executor? if so, it might be that

i wouldn't expect differences in behavior between `ploomber task` and `ploomber build` unless you're using the parallel executor

I'm not... maybe it's just a matter of how much other work my computer is doing at the time? It did happen a few times over the past couple days :thinking_face:

anyway, thanks for the pointers. I'll try to keep an eye out for anything more causal going on

Maybe if it's compute heavy the cores get busy context switching.. It'd be interesting to hear if that happens again

I could raise an exception, but I'd rather do something like `sys.exit()`

yeah, this is an undocumented feature. you can raise `DAGEarlyStop` and to signal that you wanna stop execution, then ploomber will exit with status 0

<https://github.com/ploomber/ploomber/blob/2fe474987ae6fca088e02399dbff99672a17fc95/src/ploomber/exceptions.py#L83>

but it wont work on papermill, since it masks the exception and throws its own, it has to be a Python function task

<@U0164GZJ5EK>, it definitely worked. I tested in a job running in production today. You are correct, it needs to be in a function. I added an `on_finish` step and it worked perfectly! No more failed job notifications. Super cool trick. Thanks!

you're the second user to implement this (first one was me lol) :tada:

unfortunately, soorgeon does not support Python 3.6, you have to upgrade to 3.7 or newer

hi, yeah this has been in our backlog for a while.

I'll add it to our  next sprint :slightly_smiling_face: if you have some time, PRs are welcome

unfortunately all i can contribute are user requests at this point:rolling_on_the_floor_laughing:

no worries, that's great! We'll get back to running the weekly polls to know what to prioritize - those have been pretty useful

This is probably a stretch but it would be cool to support an expression language in the placeholders (whether in env.yaml or pipeline.yaml). For example my pipeline generates date-based output folders. I can inject {{now}} to ensure consistency across the notebooks (avoids problems in the edge case where you run the pipeline at 11:59pm), but I have to keep applying the same .strftime format across the notebooks. Would be better if I could apply the xform in the pipeline.yaml using {{now.strftime(‘%Y%m%d’)}}.

expressions are actually supported but undocumented. we use jinja so any jinja valid expresion will work. we haven't documented it since encouraging logic in the env.yaml might cause beginners to put too much stuff there, but it's possible. the challenge with `{{now}}`  , is that we pass it as a string, so you'd need to parse it as an datetime then back to a string, which isn't ideal. what about supporting dotted paths in `env.yaml`?

```# env.yaml
now: functions.now # executes the function and assigns the returned value to now```
```# functions.py

def now():
    # your logic goes here
    return something```

I ended up just making a util function that I use to keep things reasonably DRY

hey, what Python version are you using? we'll release this in the next major update (0.20), but this will also drop support for python 3.6

Coming back to this. First, my util function was no good because it prevented me from using Ploomber's "product" feature to pass info across stages. So I tried the Jinja thing and it worked great. Happy to use that for now but I could definitely see it being useful to be able to format {{now}} for pipeline outputs. We timestamp different pipeline runs.

Second, I tried the new env.yaml params. Works like a charm. Thanks so much for adding that. My env.yaml file shrank considerably.

we had another question today that would be solved with the dynamic params thing (pass a function as parameter, the task receives the output of the function). I'll add it to the backlog, it's easy to implement

re env.yaml params: great! turned out to be a lot simpler than I thought, it was like 20 lines of code

Just to clarify, multiple upstreams per task? You can pass multiples as a list on the task. If not please elaborate?

is this what you're looking for? <https://github.com/ploomber/ploomber/issues/602>

Yeah, I mean exactly what <@U0164GZJ5EK> mentioned. Thanks

cool. thanks for your feedback. this is already high priority but we have a long backlog :sweat_smile: so I can't promise when we'll get to this but we will

Yeah good to know it's on the roadmap :smile:

I'm not worried I'll be able to use this soon since I know you guys move fast :smile:

This one? <https://hydra.cc/docs/intro/|https://hydra.cc/docs/intro/>. I wonder if other users has the same challenge.

I've heard about hydra but never used it. I think we need a bit more info here (hope other people can weigh in). Cause I'm unsure if there's potential for ploomber to improve env.yaml management or it's better to have an integration with Hydra

By unwieldy I mean when using complex configs I see I could either:
• pass every parameter in `params` in pipeline.yaml
• set config as dict and pass it in params, but this makes function signature not reflect what is actually needed

I mean maybe there is some simple middle way - one thing I imagine is to use decorators to validate that configs actually have required keys

I mean this is the "call" side, other side is "define" side, namely specifying stuff in env.yaml

this is great feedback. thanks! I agree with you, when you have tons of parameters passing them one by one is tedious. perhaps something that automatically grabs them would work? this reminds me of pytest fixtures, where you declare function arguments and then pytest passes the appropriate values. for example, if your env has:

```key: value
another: another-value
final: last-value```
and your task:

```def some_task(key, final):
    passs```
ploomber could inspect the function's signature and pass key and final (but not `another`). thoughts?

the tradeoff is that the pipeline.yaml is not explicit anymore, since we'd have to dynamically pass the params. it'd look something like this:

```task:
   - source: functions.some_task
     params: auto # special value to have ploomber inspect the function signature```

Hmm that might be useful, but I meant something else - the problem when we have tons of parameters env.yaml gets cluttered and it's harder to manage like hydra configs which can use single file for each config.
I actually tried using configs from other files but this has the problem that ploomber does not see when a config gets updated.

Other than that, it might be useful to have something like automatic serialization - I've experimented with decorator that converts specified argument to a specified dataclass - but I ran into problems with using decorated functions with ploomber

&gt;  I actually tried using configs from other files but this has the problem that ploomber does not see when a config gets updated.
you mean using configs for third party libraries? like you use hydra, and change the config but ploomber won't detect the changes?
&gt;  Other than that, it might be useful to have something like automatic serialization
we have serialization of products. is this what you're looking for? <https://docs.ploomber.io/en/latest/user-guide/serialization.html>

if you have some ideas on how to unclutter env.yamls, I'm all ears :slightly_smiling_face:

Wow, that's not what I meant, but it's really great to know this :smile:

By multiple configs I mean that let's say I have a MaskRCNN that has some config from other project with 20 fields.
Currently the way to pass these parameters to ploomber is either
• load config file in task, for example pass path to this config as parameter
• pass parameters explicitly to env.yaml or pipeline.yaml

The problem with first approach is that someone can then change the config and pipeline doesn't recognize this happened.

aaah. i got it now. are you passing configs from other projects because they're not written with ploomber?

you can use `resources_` for this: you can pass a path to a file and ploomber will look for changes there to re-run a task. the path is also accessible to the task so you can use it for loading the config

<https://docs.ploomber.io/en/latest/api/spec.html#tasks-params-resources>