No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

re DVC: given that you you use Makefiles, the interaction will be pretty similar, Ploomber will be the place to describe the workflows and you can use DVC for versioning the data. have you encounter any problems integrating Ploomber into DVC?

You want Ploomber to mark a task as outdated if a file changes? If so, you can pass the `resources_` key which will hash the contents of the file and re-run the task if the contents change, <https://docs.ploomber.io/en/latest/api/spec.html?highlight=resources_#tasks-params-resources|docs here> does this solve the problem?

excellent. thanks for quick reply. let me give it a try and get back to you if further questions.

IMG_20220316_161704.jpg

Hey! So are you trying to pass extra parameters to the volume?
Does the pipeline runs without errors before exporting?

ah, it should be `mounted_volumes` , youre missing an `s` at the end

PXL_20220316_135434308.jpg

looks like docker isn't installed in the computer

can you check? run `docker --help` in the computer

IMG_20220316_192811.jpg

yes, if you're running on sagemaker, sagemaker needs docker

are you trying to run this on kubernetes/argo?

yeah, you'd need to install docker in sagemaker, cause it needs to build the image there. you'd also need to ensure your sagemaker instance has permissions to push to an image registry

currently, this isn't supported, because Ploomber needs to store metadata at the folder level. I'm guessing you are declaring folders as outputs because the number of output files is too large? What's the inconvenience with having two folders?

because I have 16 other processes to run like Neurons networks. Each process is 3-5 steps long. Having  an output folder for each task is just not the best idea!

ok, how many outputs  is each task generating? is it variable or pre-defined?

It’s variable depending on the task. Can be 2 to 9 outputs per task

but in my folders structure, I have only 3 folders where I gather all these outputs. You can say I have 3 categories of outputs

in that case, you'd have to declare each file as a separate product in your `pipeline.yaml`  if you want to put all outputs in the same folder. otherwise, they'd need to be in separate folders. can you open an issue? this is great feedback, I think we should support this

hey, not currently. but you could call `papermill.execute_notebook`  inside the function to do so. what's your use case?

Hmm, makes sense.

I typically use functions instead of whole scripts, and was wondering if I had to use scripts.

wait but Papermill executes notebooks right?

I meant I have some tasks that are actual functions and not .py files, and I'm interested in turning their executions into notebooks

yeah, papermill executes notebooks, so if you want to execute as script as a notebook, you need to use jupytext to convert py -&gt; notebook, then execute with papermill

maybe the best way is to do some little refactoring to turn those functions into notebooks, what do you think?

basically I had several tasks in one file as functions

but maybe it makes sense to refactor some of them into running scripts

I don’t think you need two .yaml files. As you describe it, such a pipeline is already supported. Perhaps share your .yaml file or describe your pipeline in more detail, and explain where you think ploomber falls short here?

Just to clarify, you want something similar to this: <https://docs.ploomber.io/en/latest/api/cli.html#build-pipeline-partially>

Where it’s pointing to a different end?

Screen Shot 2022-03-17 at 11.37.03.png

In that case you can still use the partial build and run on D and E

great question, my personal preference is that conditionals inside a pipeline get a bit hard to maintain. however, there are a few alternatives I'd suggest:

1. partial builds can allow you to select the final node to execute, so you could so `ploomber build -p random-forest` or `ploomber build -p another`
2. alternatively, you may parametrize a task and have that parameter determine the logic to execute inside the task. example:
```tasks:
  source: script.py
  products: ...
  params:
    model: random-forest ```
then in your script.py:
```if model == 'random-forest':
  # do something
else:
  # do another thing```
3. Finally, if the logic is too different, you can create the common parts in a separate file and then use the `import_tasks_from` directive <https://docs.ploomber.io/en/latest/api/spec.html#import-tasks-from>

thoughts <@U036GRSDH9C>?

Yes that was exactly what I meant, thank you guys!

<https://ploomber-io.slack.com/files/U025MBQ6W7P/F037J8C6CSE/screen_shot_2022-03-17_at_11.37.03.png|follow-up question: >what if in his piacture, I have a task F which is common to both branches (after D and E)

Can I still choose one path over the other?

you could create two tasks that have `source: F.py` , just add a `name: some-name` to each one

if you go that route, then one task will have D as upstream and the other one E as upstream, in that case, you need to disable extract_upstream and declare upstream dependencies in pipeline.yaml:

```meta:
  extract_upstream: false

tasks:
  - source F.py
    name: f-one
    upstream: [D]

  - source F.py
    name: f-two
    upstream: [E]```

Annnn ok so it works also in this case, thats great!

Feel free to ask more it helps us improve the product and docs :slightly_smiling_face:

Just an empty file or plug in something in there? When running scaffold it creates it for you and then you can fill in what you want.

Also, if you create the file before scaffolding it won’t recreate it :slightly_smiling_face:

I mean, when you run ploomber scaffold for the first time, it prepopulates pipeline.yaml, tasks, scripts, etc.... I'm wondering if I could have my own default files get populated in there instead

for example, have a skeleton directory of default files

great question! I thought about this when working on the scaffolding feature but I didn't implement it. it's actually simple since scaffold takes a folder as parameter and then copies it. So I imagine something like
```ploomber scaffold --from-template <https://github.com/user/my-template>```
what do you think <@U0304HPMA1H>?

yes, that'd work. it'd be nice to define a local directory as well for that template (NFS share, but same difference I guess)

and it'd be nice to be able to set the default template location

so you can just type ploomber scaffold and get the same thing; I guess one could create a bash alias to do that as well, but would be cleaner I think in teh code itself

sounds good, can you open an issue so we keep track of it?

this sounds like jupyter is using another environment than what `ploomber build`  uses. my recommendation: open a notebook and execute: `import sys; print(sys.executable)` , then open a terminal but instead executing `ploomber build` , run `import sys; print(sys.executable)` . do these match? my guess it that you'll get two different values. let me know so i can help you troubleshoot this

you are right they both show different environment path. not sure why that is. I start my jupyter lab session from within my virtual environment.

resolved it. needed to install jupyter lab in virtual environment.

nice! that same issue has caught me many times :joy: glad you got it working!

you can use those .py files as notebooks :slightly_smiling_face: right click on them and then click "open with notebook", <https://docs.ploomber.io/en/latest/user-guide/jupyter.html|see here>. does this help?

Oh I thought you were talking more about the pipeline yaml no? <@U034URH93QE>

ok i run my pipeline, and now I want to interact with the executed .ipynb's. but it seems the ipynb's don't retain the session that is executed in the pipeline run? but I can't load in the data because the pd.read_csv(str(upstream([])) doesn't run outside of ploomber build. If I am missing something this effectively kills the interactivity of notebooks?

<@U025MBQ6W7P> this is a different thought than the message i sent

ah. so the ipynb files in the `product` section are meant to be read-only, if you wish to edit the source code it is better to edit the `.py` file in the `source:` section. you can open it as a notebook by right clicking.  <https://ploomber.io/images/doc/lab-open-with-notebook.png>

also, the jupyter plugin should automatically inject the upstream parameters, so the upstream thing should work. if the plugin isn't working, you can run `ploomber nb -i` to inject it manually in the `source`  files

You can also always save the output notebooks into a specific location in the product section like
`nb: output/get.ipynb`  so you have the context of your execution.

I use VSCode and understand how to use .py as .ipynb's, just to clear that. But if I reopen a .py as .ipynb I cannot run them outside of ploomber build because the cell that loads data no longer works outside of ploomber build. So I effectively need two cells to load data, one for the ploomber build, and one for individual runs to actually write the code.

```timeSeries=(pd.read_csv(upstream['createTimeSeries']['dotsTimeSeries']))```
needs to become
```timeSeries=pd.read_csv('actual/path/data.csv')```
in order to run the notebook outside of the ploomber pipeline and actually write the code.
then once I'm done writing, I uncomment the top line and comment out the bottom line?

ah, so if you use VSCode, execute `ploomber nb -i` and ploomber will inject the upstream variable :slightly_smiling_face:

then you'll be able to run stuff interactively using the .py files

ok, do I do this after the pipeline has been executed or should I type that in and then rerun the pipeline?

when you run `ploomber build`, it'll override the upstream if it exists, so it doesn't matter. you can run `ploomber nb -i` and whenever you want to run the full thing, run `ploomber build` - but if you change anything in your pipeline.yaml, then you'll need to run `ploomber nb -i` again

ok, and so after the entire thing runs I'll be able to open up a .py file and start working normally?

yeah, I'm guessing you're using vscode with .py files in percent format using the interactive mode , right?

because right now it just all runs, creates the desired outputs, but the files are clean so I can't go in and work on the files because they don't remember anything from the ploomber build run

yes, `ploomber nb -i` should do the trick, but feel free to post another question if it doesn't solve your issue

and I have to reenter that every time pipeline.yaml changes, would be nice if it just stayed?

(python) C:\Users\yosty\Desktop\Desktop_Folder\14 - git\timeSeriesDOTS\ploomber\dots&gt;ploomber nb -i
Black is not installed, parameters wont be formatted
Black is not installed, parameters wont be formatted
Black is not installed, parameters wont be formatted
Black is not installed, parameters wont be formatted
Injected celll:
    C:\Users\yosty\Desktop\Desktop_Folder\14 - git\timeSeriesDOTS\ploomber\dots\00-data\downLoadMetaData.R
    C:\Users\yosty\Desktop\Desktop_Folder\14 - git\timeSeriesDOTS\ploomber\dots\00-data\dataCollect.py
    C:\Users\yosty\Desktop\Desktop_Folder\14 - git\timeSeriesDOTS\ploomber\dots\00-data\createTimeSeries.py
    C:\Users\yosty\Desktop\Desktop_Folder\14 - git\timeSeriesDOTS\ploomber\dots\00-data\calculateNetworkStats.py
    C:\Users\yosty\Desktop\Desktop_Folder\14 - git\timeSeriesDOTS\ploomber\dots\01-timeSeries\xgboostWindow.py
Finished cell injection. Re-run this command if your pipeline.yaml changes.

Black is not installed, parameters wont be formatted
Is this of concern?

no, it's just a warning from a third-party library

the injected cell stays, but if doesn't update automatically, that's why you need to run it again if you change the product paths in `pipeline.yaml`

ah ok makes sense. and so now that I do ploomber build, it is rerunning everything because it will keep the variables and such so when I open the files I can execute the cells and it will be interactive

can this command be run selectively? Now that I am thinking of it, there could be potential memory issues with large projects, or ones with a lot of data. I could also go into notebooks now and
```del(largeDF)```
but it would also be nice to make individual files interactive like:
```ploomber nb -i -specificFile.py```


there aren't performance issues with the `ploomber nb -i` command. all it's doing is creating and extra cell with the `upstream`  variable, but it isn't loading anything. so you're good

if you mean performance issues when running all your pipeline, then you can selectively run tasks with `ploomber task {task-name}` or do a partial build with  `ploomber build -p {some-task}`

what is the proper way to stop a pipeline when it is running? I just reran everything by accident and just need to rerun one quick file.

What would be the task name here?

```  - source: 01-timeSeries/xgboostWindow.py
    product:
      nb: 01-timeSeries/xgboost.ipynb
        # resultsDict: 00-data/model_output/xgboostResults.pickle
        # pcaResultsDict: 00-data/model_output/xgboostPCAResults.pickle```

```xgboostWindow```
You can also give it a custom name, in case you want something shorter, with this key: `-name: task_name`
Also, for VSCode, if you wanna run this command automatically you can configure a file watcher.
<https://marketplace.visualstudio.com/items?itemName=appulate.filewatcher|Click here for a VSCode extension>
And here’s the docs link: <https://docs.ploomber.io/en/latest/user-guide/editors.html>

re task name: `xgboostWindow` it's always the filename without the extension

I am getting this error KeyError: "DAG does not have a task with name '{xgboostWindow.py}'"
I've tried
01-timeSeries/xgboostWindow.py
01-timeSeries/xgboostWindow
xgboostWindow
xgboostWindow.py

with ploomber interact and list(dag) I see this:
```list(dag)
Out[1]: 
['downLoadMetaData',
 'dataCollect',
 'createTimeSeries',
 'calculateNetworkStats',
 'xgboostWindow']```

ok at least now I get
ploomber.exceptions.TaskBuildError: Cannot build task 'xgboostWindow' because the following upstream dependencies are missing: ['calculateNetworkStats', 'createTimeSeries']. Execute upstream tasks first. If upstream tasks generate File(s) and you configured a File.client, you may also upload up-to-date copies to remote storage and they will be automatically downloaded

oh yes. so you need to build the dag first, then you'll be able to run tasks individually

ok so I check this afternoon and one of my tasks failed with partial execution, but when I go to open the ipynb to troubleshoot, its not interactive so its hard to de-bug. I thought it would be interactive now?

Or its just letting me inject the locations so I can now rerun the notebook until the error? Ok yes sorry I get it now.

you can enable multiprocessing for running pipeline tasks in parallel in your pipeline.yaml
```executor: parallel```
if you wish to use multiprocessing.pool inside a task that's also possible

image.png

ok thank you, a few of my files have multiprocessing in them.

let's say I have a pipeline that looks like this, how could I get each red box to be able to execute at the same time?
say, allow groups (1,2) (3,4,5,6) (7,8) to run in parallel

The parallel executor will do that for you. However, there is no way to enforce that certain tasks run at the same time, the parallel executor will create a pool of processes and schedule new tasks as others finish 

For example. It task 1 takes an hour to finish but 2,6,8 are pretty fast, the pool still has to wait for 1 before it can continue 

`(market_watch) aiujdm2 market_watch (main) &gt;&gt; ploomber build`
Loading pipeline...
Traceback (most recent call last):
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/cli/io.py”, line 37, in wrapper
    fn(**kwargs)
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/telemetry/telemetry.py”, line 551, in wrapper
    raise e
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/telemetry/telemetry.py”, line 537, in wrapper
    result = func(_payload, *args, **kwargs)
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/cli/build.py”, line 54, in main
    dag, args = parser.load_from_entry_point_arg()
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/cli/parsers.py”, line 215, in load_from_entry_point_arg
    dag, args = load_dag_from_entry_point_and_parser(
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/cli/parsers.py”, line 488, in load_dag_from_entry_point_and_parser
    dag, args = _process_file_dir_or_glob(parser)
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/cli/parsers.py”, line 440, in _process_file_dir_or_glob
    dag = DAGSpec(dagspec_arg, env=env).to_dag()
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/spec/dagspec.py”, line 199, in __init__
    self._init(data=data,
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/spec/dagspec.py”, line 331, in _init
    tags_other) = expand_raw_dictionaries_and_extract_tags(
  File “/Users/aiujdm2/.local/share/virtualenvs/market_watch-dtlP-L11/lib/python3.9/site-packages/ploomber/env/expand.py”, line 44, in expand_raw_dictionaries_and_extract_tags
    zip(*[
TypeError: ‘NoneType’ object is not iterable

When i open .py in jupyter as notebook, i do not code cell insertion for products either.

are you using the `import_tasks_from` directive?

if so, it might be that the file you're using is empty. e.g. `import_tasks_from: something.yaml` and something.yaml is empty

yes. I have to comment out code there as that task in under progress.

Alright. It's breaking because it's empty. I'd recommend commenting out the import_tasks_from line, let me know if that fixes it 

sure. thanks for the feedback, I'll make sure we validate the input so we provide a clearer error message :slightly_smiling_face:

so you have a pipeline param and you want to ignore certain branches if it has certain value? is the parameter something you put in your `pipeline.yaml` or some runtime parameter computed inside a task

&gt; so you have a pipeline param and you want to ignore certain branches if it has certain value?
correct

&gt; is the parameter something you put in your `pipeline.yaml`
correct

unfortunately, we don't have a way to do so. but you can implement a wrapper using the Python API. essentially grab the parameter , and then call `dag.build_partially` depending on the value. check out this example: <https://github.com/ploomber/projects/tree/master/cookbook/python-load>

sure, feel free to ask any other questions if you encounter issues implementing this, but hopefully, it'll be straightforward

one thing I noticed, in the example, the function returns a DAG object, in your case, you want to call `dag.build_partially` so you wont be able to use the CLI, but rather something like `python pipeline.py`  to execute the pipeline

yep, time to learn the python API :slightly_smiling_face:

not currently, but this makes sense. can you open a github issue? as a temporary measure, you can switch parameter from the command line, so you could write a bash script to call `ploomber build --env-param value --env-param2 another-value`  to see how to switch your parameters, run `ploomber build --help`

<https://github.com/ploomber/ploomber/issues/679|created the issue> :slightly_smiling_face:  did my suggestion help?

it did in my case where only 1 or 2 parameters were different. Thanks for creating the issue.

I’m finding using --env-&lt;param&gt;, --entry-point, and -p  are very useful options when in active development  . The export to system environment variable is tricky as it sets those value for entire session which is not desirable when we are switching back and forth.

thanks for your feedback! We'll address this issue soon. but of course, PRs welcome :slightly_smiling_face:

is this task a python task with SQL on it? or a purely SQL task? If the former, you might add an `if` statement that runs the SQL code or not, depending on the parameter. would this solve the issue?

ok, this is doable by adding some <https://docs.ploomber.io/en/latest/cookbook/spec-load-python.html|extra Python logic> - but I need to run some experiments, because the `-f` flag may interfere. we got similar questions already so I'll draft an issue/proposal so we address this, will post the github issue in the <#C026JAEGCTG|> channel later

I think altair plots wont show up until you open them in jupyter - that's bc of the security mechanism to prevent <https://jupyter-notebook.readthedocs.io/en/stable/security.html#our-security-model|arbitrary JS execution>. re multiple products, <https://github.com/ploomber/ploomber/issues/673|there's an open issue>, please add a comment or reaction so we know you are interested and we can prioritize

On the 2 products, you can still do it by duplicating the task and having 1 to output an ipynb and the other to generate an html.

Hi Mitch, thanks for your feedback! As you can see, we've been getting this a few times already.  So I think we should do something about it. I need to give it some thought to ensure the solution is general enough, seems like there are a couple use cases. I'll draft something tomorrow based on existing feedback, and will post the GitHub issue here so others can weigh in

yes, you can use the same `source` many times, but you have to turn automatic upstream extraction off:

```# pipeline.yaml
meta:
  extract_upstream: false

tasks:
  - source: task.py
    name: task-a
    products: ...
    upstream [another]

  - source: task.py
    name: task-b
    products: ...
    upstream [a-different-one]```
in your case, since `drop_columns` will have different upstreams you can refer to them without knowing their name like this:
```def drop_columns(upstream, product):
    train = pd.read_csv(upstream.first["train"])
    holdout = pd.read_csv(upstream.first["holdout"])
    # cleaning....```
does this solve your issue?

(note: if you turn extract_upstream off, all tasks must have an upstream key)

Ahm, not so sure that solves my problem though…

I still have to hardcode `"train"` and `"holdout"` in my Python code…

I was thinking of something to the tune of:

```tasks:
  - source: task.py
    name: task-a
    products: ...
    upstream [another.train]

  - source: task.py
    name: task-b
    products: ...
    upstream [another.holdout]

  - source: task.py
    name: task-c
    products: ...
    upstream [another.some_other]```
And then Python
```def drop_columns(upstream, product):
    generic_df = pd.read_csv(upstream)
    #code...```

yeah, so in `drop_columns` you can do `upstream.first` and that's going to return the products from the upstream task (whatever that is, since you're not referring the upstream by name).  what's missing with this approach?

So I could be specific in the `upstream` key in the pipeline:
```tasks:
  - source: task.drop_columns
    name: task-a
    products: ...
    upstream: [another.train]```
And then in `task.py`:
```def drop_columns(upstream, product):
    generic_df = pd.read_csv(upstream.first)```
`upstream.first` would contain the product `train` of the `another` task?

ah ok, i think I see where the confusion is coming from. in `upstream` you put the name of the upstream task. then `upstream.first` returns the product of that upstream. example:

```tasks:
  - source: tasks.train
    name: train
    products:
      a: ...
      b: ....

  - source: tasks.drop_columns
    products:
      c: ...
    upstream: [train]```

then `upstream.first`  inside `tasks.drop_columns` is the same as doing `upstream['train']` hence, it returns a dictionary with keys `a` and `b`, since those are the products of `train`

Oh, still not quite my use case. If I understand correctly, I would still need to reference `a` and `b` in my Python code, as in `upstream.first["a"]`, right?

I want to avoid having to reference a specific upstream product within my Python code. The way I was thinking is to be able to directly specify where the value of `upstream` comes for a given task.

Kind of:
• for task *a* with source *drop_columns*, its upstream is *split.holdout*,
• for task *b* with source *drop_columns*, its upstream is *split.train*
• for task *c* with source *drop_columns*, its upstream *is fetch_new_data.data*
and then have the function `drop_columns` be as generic as possible:

```def drop_columns(upstream, product):
    df = pd.read_csv(upstream)
    # ...```

ah ok, got it. interesting use case, never thought about it. right now, ploomber pushes all products downstream, so what you're suggesting isn't supported, I'll open an issue since this is an interesting use case. I think for now, what I suggested can help you. then, you have two options. 1) either standarize your products (e.g. tasks product a single file or produce a dictionary with the asme keys) or 2) add a bit of logic so you don't need to hardcode the product key. e.g if they generate a dictionary with a single key:

```def drop_columns(upstream, product):
    key = list(upstream.first)[0]
    df = pd.read_csv(upstream.first[key])```
if the upstream generates a single product (instead of a dictionary), then it's simpler:

```def drop_columns(upstream, product):
    df = pd.read_csv(upstream.first)```

interesting request. So if i understand correct, <@U029QTPAQ8H> your drop task still has to have explicit knowledge of two upstream products as you are producing respective cleaned output for each. how would you be able to generate different named output without this understanding?

<@U0164GZJ5EK> cool, I’ll keep an eye on the issues then.

<@U0370FZQZ7E> the task, as an abstraction (in the pipeline definition) yes, has to know about the upstream. However, at code (or using Ploomber terms, source level) I see no need for it to hold this knowledge.

Just wondering if instead of generating named output, can you define product as a direcory instead of a file.  In downstream task you just then have to get all files from this folder and apply drop task on them.

Sure I could workaround in many ways, however they all sound hacky. To me, it would make more sense to allow true task reusability by letting users specify connections amongst them in the pipeline file, rather than hardcoding them within the source code itself.

The more I think about this, the more it makes sense to me...

For example, I was thinking of having three pipelines:

One for initial training, other for subsequent retraining and one for batch inference. Reusing the same code for feature transformation across three pipelines would be great, reusing the training/evaluation/validation code for the initial and subsequent retraining would be great.

At the moment I see this only being possible by creating thin wrapper methods.

for training &lt;&gt; serving, we have a `import_tasks_from` feature, that allows you to put all feature engineering code in a file and then have a `pipeline.train.yaml` and `pipeline.serve.yaml` files (<https://docs.ploomber.io/en/latest/deployment/batch.html|see here>) - but I agree, there's more we can do to enable this modularization

hi, i created an issue. I think this should fix your problem <@U029QTPAQ8H>, feel free to comment any feedback. <@U0370FZQZ7E>, please also let me know what you think. I'd like to open the discussion so we add a new example showing how to re-use tasks <https://github.com/ploomber/ploomber/issues/682>

Looks like there is some incompatibility with pygraphviz. Can you see if this fixes the problem?   

<https://github.com/ploomber/ploomber/issues/671#issuecomment-1074688313|https://github.com/ploomber/ploomber/issues/671#issuecomment-1074688313>

oh, interesting, so if I run it in a different directory, it works

is there some sorta caching you're doing?

weird. we're not doing any chaching. what version of python and graphviz ( `dot -V`) are you using? I'll try to replicate it

dot - graphviz version 2.50.0 (<tel:202112090339|20211209.0339>)

oh, interesting... ok, I think I figured it out

I then run a particulra conda environment using a jupyter kernel

it's supposed to run everything in that environment

but it looks like graphviz isn't being repsected.

also, when I drop to %%bash, it somehow drops out of the conda environment as well... i image its related

interesting. So I guess you have two environments, right? Could it be the case that when running
```pip uninstall ploomber
pip install git+<https://github.com/ploomber/ploomber@colors>```
and then test the plot, you installed in one but tested in another? if you get a chance, please install from git, then run `ploomber --version` and check that you see a `dev` at the end (this means it was installed from git), then check if `ploomber plot` works.

just do `PLOOMBER_ENV_FILENAME=&lt;name of env file&gt; ploomber ...`

I assume you mean `PLOOMBER_ENV_FILENAME=&lt;name of env file&gt; jupyter …`

oh I misinterpreted your question. Maybe you're after this <https://docs.ploomber.io/en/latest/user-guide/jupyter.html#custom-jupyter-pipeline-loading>

Indeed, that seems to be what I was looking for

Screen Shot 2022-03-28 at 7.32.04 PM.png

Screen Shot 2022-03-28 at 7.28.02 PM.png

try with `Path('some_tasks.yaml').write_text('')`  to create an empty file. does that change the error message?

also, please send the full traceback from pytest

Screen Shot 2022-03-28 at 7.44.52 PM.png

I tried that edit, but it seems I encounter the same error. I have attached the full stack trace

ah ok. I think it's because the DAGSpec is initialized from a dictionary. instead of a path to a file. add a new line before the `spec = DAGSpec(spec_d)`  to store the contents of  `spec_d`  in a `pipeline.yaml` file, (using yaml.dump), then replace the `spec = DAGSpec(spec_d)` for `spec = DAGSpec('pipeline.yaml')`

Screen Shot 2022-03-28 at 8.43.57 PM.png

Would it be like this? When I run this again I get the same error

please push that to your fork. I'll take a look :slightly_smiling_face:

just pushed it, thanks! sorry for all the trouble

what's the use case? are you appending to a table or creating a new one?

the best pattern here is to add an <https://docs.ploomber.io/en/latest/cookbook/hooks.html|on_finish hook> to the task that produces the data. inside the hook, you can call the mongodb driver and upload the data. alternatively, you may embed the uploading logic within the task itself, but I think a hook is cleaner

Right now it seems that I have to choose one or the other, and then if I want to switch I have to restart Jupyter.

yes, as you mention, the only way to switch the jupyter integration is to restart the instance which isn't great. I've been thinking that we need to introduce some sort of "ploomber config file" at the project level (since a single project may have more than one pipeline), inside that file one could specify multiple pipeline.yaml for the jupyter integration to work. thoughts?

the same file could be used for other project-related config stuff

The consolidated config is a good idea. That would help with another issue I’m seeing, which is that I’m having to duplicate certain config across multiple env.[name].yaml files. I have dev and prod env files, but certain config is common to those, such as the list of time series for which I’m training models.

what you mention about env.yaml already came up. is this what you're looking for? <https://github.com/ploomber/ploomber/issues/679>

For context, I want to use notebook files in VS Code. So formatting as `.py` files, and using `ploomber nb -S` does not help me

in case it's useful:

```$ ploomber --version                                                                                                                   
ploomber, version 0.17```

ah. that sounds like an error when parsing the notebook. I think I've seen it before, should be an easy fix. can you open an issue?

<https://github.com/ploomber/ploomber/issues/692>

awesome, thanks for the feedback, will take care of this! :slightly_smiling_face:

you're right, both are valid use cases :slightly_smiling_face: it makes sense to organize in dev/prod with multiple env.yaml. that section explains how the env.yaml resolving logic works. if you have a `pipeline.yaml` then it looks for an `env.yaml`, if you have a `pipeline.something.yaml`, it looks for an `env.something.yaml` but it falls back to the `env.yaml` - i agree the training/serving isn't a great example, because training and serving are likely to need more less the same parameters. this example use case applies when you have two pipelines with not so much overlap

OK, thanks. Yeah I can imagine cases where you might want to split it up by pipeline stage. But to me it seems like the environment use case is going to be more typical -- especially since the training stage and the serving stage are presumably going to use the same model store. So from a documentation point of view it seems clearer to prioritize that use case over the per-stage use case. Anyway take it as a data point from one user.

Incidentally this feeds that "shared env.yaml" thing we were talking about yesterday -- even if I wanted to split my env.yaml up across training and serving, for example, I'd still want to define the model store location/credentials in one place

yep! this "shared env.yaml" has come up a few times now :slightly_smiling_face: we'll add it to our sprint for next week. ofc, PRs always welcome!

Is the traceback visible or does it only show the error message?

soorgeon refactor ../SHNetTests.ipynb  --df-format parquet
Added 'output' directory to .gitignore...
Error: An error occurred when refactoring notebook.

Try refactoring as a single task pipeline:

$ soorgeon refactor ../SHNetTests.ipynb --single-task

Error details:

unsupported operand type(s) for +=: 'NoneType' and 'str'
(automl) [arm2arm@nnewl3 pipeline]$

so my error management code is hiding the traceback :confused: let me try to debug. I think i know where the error might be. thanks for your feedback! will update you

thank you, i wondering if it is possible to use dask parquet output. i can see my all tasks dataframes are dumped as a pkl single files

yes, pkl is only the default, but you can manually change the code to store/load the data frames after running `soorgeon refactor` , the `--df-format`  does it automatically for variables with the `df_` prefix. I'll look into this now :slightly_smiling_face:

ok, so I published a new soorgeon version to ease debugging. please upgrade (ensure you get version `0.0.15`  (`soorgeon --version`) then run `soorgeon refactor ../SHNetTests.ipynb --df-format parquet --log error` - this should now print the traceback, to help me understand why the `--df-format parquet` thing is breaking

What’s in pickle.dumps(model) ?
Does it save it when it’s not parameterized?

maybe try pickle dump? since you’re writing bytes and use dumps
Something along these lines:
```with open(product['model'], 'wb') as f:
    pickle.dump(clf, f)```

ok, will try it in separate project to check what's going on.

Yeah, this is a pickle problem. Independent of Ploomber, my recommendation is to check the docs in the tpot library, most likely, your model is stored in one of the attributes. For example in scikit learn, you get it using “model.best_estimator_” - you may also try with other pickling libraries like joblib or cloudpickle 

Can you try installing this required mistune version? I just tried in conda and it works fine which leads me to think maybe the env has some extra dependencies.

Also what's the error it gives on scaffolding?

```AttributeError: module 'mistune' has no attribute 'BlockGrammar'```

I guess it is due to the version mismatch

Ok, then I try conda. Do you happen to have a list of specific versions I should use for all the dependencies?

Care to open an issue?
For conda just follow this it should set you up: <https://docs.ploomber.io/en/latest/get-started/quick-start.html#conda|https://docs.ploomber.io/en/latest/get-started/quick-start.html#conda>

<@U039ACSR0HG> I assume it worked out? :slightly_smiling_face:

I haven't tried it yet. Because I am still checking how that will affect my development environment. So, my plan is to use PyCharm as the IDE, and I am imagining that I would need to include ploomber and all its dependencies in the dev requirements file (please correct me here if I am mistaken). Since I would want to run all the ploomber commands from within PyCharm. And so far, I only setup virtualenv projects on PyCharm. So, I am investigating  how it would be if I use Conda with PyCharm

If I don't get any further, I will create an issue

Thank you for your help so far :slightly_smiling_face:

Awesome! So for PyCharm we have this Doc : <https://docs.ploomber.io/en/latest/user-guide/editors.html|https://docs.ploomber.io/en/latest/user-guide/editors.html>

Also, I'm working with PyCharm, for the conda env you just need to reference it as the interpretor path and it will add it. And for the requirements, yes you're right a reqs.txt is recommended :blush:

I've encounter this issue when using old versions of pip, the resolver sometimes fails to install the right versions. Can you try from a clean virtual env, upgrade pip then install Ploomber? 

Alternatively, you may want to force downgrade of mistune: pip install “mistune&lt;2”

So, I decided to stay with venv, since I am more used to sharing venv settings with other team members through git.

cool! glad it worked! don't hesitate to send any other questions!

I also did some modifications to the requirements files (don't know if they will break later or not since my pipeline is still empty but I will fix any issues when they come up). I rearranged the requirements to be as follows:
requirements/base.txt
requirements/local.txt
requirements/production.txt

in my base.txt, I have:
```ploomber
pandas
scikit-learn```
