No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

Is this for committing code? If yes, it’s better you start with good first issues. There’s no plan atm to have a cli only interface for the telemetry.

please check this out <@U033LDTP09Y> :slightly_smiling_face: feel free to post comment on any of them so we assign it to you <https://github.com/ploomber/ploomber/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22>

I would like to try this if this is unassigned <https://github.com/ploomber/ploomber/issues/524|https://github.com/ploomber/ploomber/issues/524>

Any other that you are interested? That one has been taken 

I can try <https://github.com/ploomber/ploomber/issues/429|https://github.com/ploomber/ploomber/issues/429>
I guess it is unassigned.

Another one deepcopy dag error message improvement <https://github.com/ploomber/ploomber/issues/571|https://github.com/ploomber/ploomber/issues/571>

Sure. Please comment on them so I can assign them to you 

You can also go over the contributing guide and start a pr, doesn’t look like these are assigned :slightly_smiling_face:

Hi, added test case for issue#429, let me know if there are any comments

Hi <@U0164GZJ5EK>, I'm working with ploomber on windows machine now and I'm getting following error
`------------------------------------------------------------------------------- C:\dev\python\contrib\templates\ml-basic\fit.py --------------------------------------------------------------------------------` 
```Traceback (most recent call last):
  File "c:\dev\python\contrib\ploomber\src\ploomber\tasks\abc.py", line 561, in _build
    res = self._run()
  File "c:\dev\python\contrib\ploomber\src\ploomber\tasks\abc.py", line 670, in _run
    self.run()
  File "c:\dev\python\contrib\ploomber\src\ploomber\tasks\notebook.py", line 573, in run
    self._converter.convert()
  File "c:\dev\python\contrib\ploomber\src\ploomber\tasks\notebook.py", line 139, in convert
    self._from_ipynb(self._path_to_output, self._exporter,
  File "c:\dev\python\contrib\ploomber\src\ploomber\tasks\notebook.py", line 195, in _from_ipynb
    path.write_text(content)
  File "c:\program files\python38\lib\pathlib.py", line 1256, in write_text
    return f.write(data)
  File "c:\program files\python38\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\ue6c6' in position 237531: character maps to &lt;undefined&gt;```


ah, yeah. I've seen it before. can you open the  `pipeline.yaml` ? you'll see a path that has a  `.html`  extension. change it for ipynb

there's some incompatibility with one dependency on windows

Yes, it's working now.
Another one `FileExistsError`: (expected behavior overwrite right?)
```FileExistsError: [WinError 183] Cannot create a file when that file already exists: 'C:\\dev\\python\\contrib\\templates\\ml-basic\\output\\nb.ipynb' -&gt; 'C:\\dev\\python\\contrib\\templates\\ml-basic\\output\\nb.html'```
Steps to reproduce:
1. ploomber build example
2. update `pipeline.yaml` product nb file extension
This error popped up when I updated `pipeline.yaml` file

thanks for reporting this, can you open an issue?

Hi, please ignore commit messages mentioned in Issue #571 I removed auto mention of issue id in my commits now

Hi <@U0164GZJ5EK>, made changes as per comments but not sure why CI macOS taking time. It happened yesterday as well. Here is the link for actions tab
<https://github.com/ploomber/ploomber/runs/5363227128?check_suite_focus=true>

yeah, I noticed the same thing with one of my builds. some tests require R so we install it as part of the CI, looks like the server where we pull R isn't responding reliably. no worries, we can ignore it

i'll review the PR later today, thanks for your contribution!

Hi <@U0164GZJ5EK> / <@U025MBQ6W7P> any thoughts on *Sankey charts* related to issue <https://github.com/ploomber/ploomber/issues/540|#540>: replace dependency on pygraphviz?
Also, according to the article author wants us to use js based solutions one of them is d3-dag. This is js integration as far as my understanding and research.
Let me know if you have any other understanding or expectations.

It requires either npm package or html template as per this <https://erikbrinkman.github.io/d3-dag/|doc> at the moment graphviz is handling and rendering output for us handling all the intermediate steps

can we bundle the package? maybe if the user does `ploomber build` we could download the library and call it to generate the diagram?

and once it's local, then we don't need to download again

i guess this is all we need
```<https://unpkg.com/d3-dag@0.9.0>```


yes, this can be used for html template removing npm dependency. We have to write DAG building logic to be passed to template.
I'm yet to test this for small poc and following are expectations order wise (previous is mandatory for current step):
1. Bundling library to check on install, code is rendering some dummy graph
2. Dynamic graph generation through Python dag builder logic
3. Check for available customization options (wrapping/ sizing nodes and any *on hover* functionalities on node and at least better or almost similar representation to existing plots) 
do share additional inputs/ articles if you've any.

sounds good. thanks a lot for working on this! asking users to install pygraphviz has been a big pain point since the early days and this will have a big impact on everyone!

image.png

out.html

Hi <@U0164GZJ5EK> &amp; <@U025MBQ6W7P>, I tried steps 1 and 2 for small poc and attached image is the output and along with html output file and function that generates it.
For python to interact with templated html we require jinja2 dependency.
I'm currently exploring on d3js stack for the dynamic graph generation in required format and progress so far made is based on tweaked examples.
```def gen_plot():
    print("executing gen_plot")
    template= jinja2.Template(open('./assets/index.html').read())
    updated = template.render({
        'custom_heading': "Python generated",
        'json_data': open("./assets/data-dag.json").read()
    })
    with open("out.html", 'w') as f:
        f.write(updated)```

this looks great! Thanks a lot for working on this. we already use jinja2 so that's fine

<@U033LDTP09Y> This is great!
I saw that in the PR.
Is there a way to print the products of the tasks as well? Does the colors on the paths mean anything?
Also the color are still indicating the execution status right?

Hi <@U025MBQ6W7P>, the color on the paths were just random ones and doesn't mean anything and shared is not a dag that got generated from ploomber but with same random dynamic data passed. Feel free to share any thoughts if you have on this.
PS: got busy lately, will update on this item once I make progress.

Oh so it’s just a sample from the D3 directory? I thought you had it working with a ploomber pipeline

No worries.
When you do, make sure to print the products of the tasks as well?
And validate that the colors are indicating the execution status?

Hi, please find the sample in the git link for the d3dag
<https://github.com/Vinay26k/dag-visualization/tree/gh-pages>
Code: <https://github.com/Vinay26k/dag-visualization/tree/d3dag>
Direct link (has both code base link + demo page): <https://vinay26k.github.io/dag-visualization/>

For sample I have taken the ploomber plot ml-basic dag and added code for the same to parse it through dag.py script.
Command executed to generate dag data from ploomber =&gt; `print(nx.readwrite.json_graph.node_link_data(G))`

awesome! can you start working on adding this code to the repo? Let me add a few pointers to the github issue, and we can discuss there

thanks for the question <@U034EKU4BC0>! you can check out <https://ploomber.io/blog/survey/|the survey> we did where we include Kedro. A few things that come to mind (please correct me if I'm wrong on any of these points, since I'm not a kedro user and It's been a while since I tested it):  I've seen some Kedro examples, and my personal take is that they contain too much boilerplate in the pipeline definition (our integration with Jupyter/VSCode/PyCharm allows data scientists to build pipelines without boilerplate in a much leaner way), the support for production platforms limits to showing a script that turns a kedro pipeline into something like an Argo pipeline (so essentially you need to copy paste that thing) - on the other hand, we have a <https://github.com/ploomber/soopervisor|separate package> that streamlines this process, I dont think they have incremental builds (caching previous computations so the next time you run the pipeline, only tasks that changed executed), we have native support for SQL, so you can create tasks from .sql files instead of creating a .py that runs the query (we manage the connection to the db for you), this simplifies the pipelines a lot (<https://github.com/ploomber/projects/tree/master/templates/spec-api-sql|here's an example>), <https://github.com/ploomber/projects/tree/master/templates/spec-api-r|we also support R>.

may I ask what are the upsides and downsides you've seen in Kedro?

Just to add on the above, sometimes kedro is a lot of overhead and can be an overkill to rapid experimentations, there’s a lot to manage compared to a pip install.

I have read the survey.

• Yes boilerplate is what bothers me in kedro. 
• Integration with jupyter notebooks sounds like a terrible idea to me, but that's ok, I will just ignore the feature :]
• I don't have (yet) a lot of experience with deploying kedro pipelines. I just figured from the docs that both kedro and ploomber supports aws batch which is something we might eventually use (as we use aws for all our services)
• Caching in kedro is a thing. You can run pipelines from a specific node/task. This is the same in ploomber if I understand correctly.
• SQL in tasks got my attention. Although often the SQL scripts we use are dynamically constructed in python and called via pandas.
Downsides of kedro that come to mind right now:
• boilerplate
• supports only dags (ploomber too currently)
• I really like using # %% + scientific mode in pycharm when developing a code and later moving it into functions. In kedro, this does not work that well, as the inputs to a node/task are outputs of some node in the pipeline. A workaround is to call all the tasks by yourself and handle the correct routing of the inputs/outputs yourself. What I tend to do is to stop the pipeline in a debugger a basically develop my code there, which sounds probably worse than it actually is. I am not sure, however, how much this can be improved in a pipelining tool.
Upsides:
• while it makes code harder to write, it makes it easier to read/navigate the project.
There are definitely more things but these are those that came to my mind first.

We will be starting a new project next week and my plan is to pitch ploomber and give it a shot and than compare the hands on experience.

<@U025MBQ6W7P> overhead from what exactly? The most overhead I had using kedro is because I try to avoid caching and often rerun the whole pipeline.

• Give the jupyter integration a shot :sweat_smile: - we allow users to use the `# %%` format and open it as notebooks in Jupyter, so the same file can be developed in Jupyter/VScode without the complexities of the ipynb format, and to check that thing isn't broken you can just call `ploomber build` and it'll orchestrate execution
• you mention that kedro only supports DAGs, can you expand on that? What features are missing?
• yeah, using the `# %%` is very convenient for data scientists because they can still develop interactively but since we simplify the modular pipeline building part, they can create a lot more maintainable code (10 tasks, 20 cells each instead of a big script with 200 cells)
• we support templated SQL, <https://docs.ploomber.io/en/latest/user-guide/sql-templating.html|check this tutorial> - same concept as generating sql from Python but a lot simpler. while it is not as powerful as using Python, it covers 90% of use cases and simplifies the code a lot
please share your experience when you give ploomber a try. and don't hesitate to post any questions, we are happy to help!

"you mention that kedro only supports DAGs, can you expand on that? What features are missing?"

Well, cycles. If I would like to get low level and run for example some optimization loop as pipeline of task/nodes this would  not be currently possible either in kedro or ploomber.

 SQL templating looks interesting, I will look into it. (We have tried <https://pugsql.org/> as an option between form and plain SQL, but there were some issues I can't exactly remember right now - something regarding performance i think)

re cycles: ah, we've been chatting about this a couple times with other members of the community. since this keep coming up, I think we should work on it :slightly_smiling_face: - just to ensure I get this correctly. you want a cycle that will exit when some condition is met (e.g. model performance passes some threshold)?

Yes. I probably want to call a user-defined function, that decides if the loop should stop or not.

cool. we actually have an open issue about this so feel free to share your thoughts <https://github.com/ploomber/ploomber/issues/474>

<@U0164GZJ5EK>, apologies I just noticed that there was request for feedback from early January. I will include some.

<@U0164GZJ5EK> Does ploomber supports any inputs/outputs "routing"?

In kerdro I can have a following pipeline:
```def create_pipeline():
    node1 = node(func=node1_func, inputs="a", outputs="b")
    node2 = node(func=node2_func, inputs="c", outputs="d")
    node3 = node(func=add, inputs=["b", "d"], outputs="sum")
    return Pipeline([node1, node2, node3])```
Just by looking at the pipeline, I can see how are the outputs passed through the pipeline. I can for example split a dataset in a node into training and testing sets, and just by setting the inputs and outputs ensure, that node/task for model training receives only the training set.

From that I understand, in ploomber you specify only the order in which the tasks should be evaluated (using upstream)?

```  - source: add.py
     name: add
     upstream: [b, d]```
or perhaps
```- source: node3
     name: add
     upstream: [node1, node2]```

You got it <@U034EKU4BC0> : when you set the upstream relationship, you are implicitly saying that the outputs of one task become inputs of the next one. No need to route the outputs 

IMO, setting order of execution and inputs/outputs is redundant. Ploomber will pass the paths to all the inputs available, if you don't need some of those, then that's fine. You can ignore them 

For our use cases it is not redundant, it is a feature  to consume only part of the outputs.

Yeah, Ploomber leaves that up to you. So say you have A-&gt;B, and you have outputs A1, A2. Ploomber will pass the paths to both A1 and A2 to B. And your code in B can decide if it loads one of those or both. There is no need to tell Ploomber what you wanna do in B

feel free to ask us anything in the <#C028KJCJDRT|> channel!

This is my current code, which works but does not run in parallel.
```for f in files:
    env = make_env(f)
    dag = DAGSpec('pipeline.yaml', env=env).to_dag()
    dag.build()```

You got it, grid allows you to parallel on the task level. So in <https://docs.ploomber.io/en/latest/cookbook/grid.html?highlight=grid|this example> we see multiple tasks from a single task (via paramaterizing the estimators)

You also have a reference there for the docs, is that what you were looking for?

If it’s about notebooks we have an open issue on it, feel free to share thoughts there.

I'm not sure. I'm confused on how to calculate outputs, since they are unique to each input file.

how does your dag look like? is it like a straight line? a -&gt; b -&gt; c?

yeah. the way to go is to build one big DAG, however, we currently do not have a way to merge many dags into a single one (so you can run them in parallel). can you open an issue on github?

the only way to do this at the moment is by using the Python API directly. you may want to go that route <https://github.com/ploomber/projects/tree/master/python-api-examples>

i meant, not using DAGSpec, but using the DAG object directly, like this: <https://github.com/ploomber/projects/blob/master/python-api-examples/examples/basic.py>

Making one big DAG is fine by me, I'm just not sure the proper way to accumulate all the tasks

I guess my question still remains. I will look at what I can do with a DAG object

yeah, please open an issue - I quickly took a look at the code and I think this change would be trivial to implement

if you want to write your pipeline with the Python API, check the example i sent. then you'll need to create ShellTask and File

<https://docs.ploomber.io/en/latest/api/_modules/products/ploomber.products.File.html#ploomber.products.File>

<https://docs.ploomber.io/en/latest/api/_modules/tasks/ploomber.tasks.ShellScript.html#ploomber.tasks.ShellScript>

for your ShellScript:
```from pathlib import Path

ShellScript(Path('my-script.sh'), product=File('some-output')) # initialize with a Path object```

Don't I need to pass a `DAG` to `ShellScript` ?

```from pathlib import Path
from ploomber import DAG
dag = DAG()
one = ShellScript(Path('my-script.sh'), product=File('some-output'), dag=dag) 
two = ShellScript(Path('another-script.sh'), product=File('another-output'), dag=dag) # initialize with a Path object
one &gt;&gt; two
dag.build()```

not sure if that actually runs but that's the idea haha

you can create a loop like:
```dag = DAG()

for file in files:
   # add tasks for every file you want to process here```


oh and to run this in parallel:
```from ploomber import DAG
from ploomber.executors import Parallel

dag = DAG(executor=Parallel())```

I get an error saying "ShellScript must include {{product}} in its source" when I try to make a new ShellScript

are you passing a path object? `from pathlib import Path; ShellScript(Path('something.sh'))`

no worries, it's bad design on our end, we haven't taken the time to fix it

OK great let me try. I do like Path objects

Great thanks let's see if I can extend this pipeline, maybe add some followup tasks

nice, feel free to post any other questions

I have to ensure each `ShellScript` has a unique name, which is behavior I can agree with. Thanks again. have a good weekend

What kind of tests are you thinking of? Unit testing functions defined inside the notebook or just smoke testing that the notebook runs?

<@U0164GZJ5EK> Ack, sorry, just saw this.  Was thinking Smoke Tests, yeah!  Doublechecking that an upgraded package, or a change we made to an internal library, didn't break the notebooks.

this post will be useful, covers testing + other things: <https://ploomber.io/blog/nbs-production/>

my recommendation is to run them often with a sample of the data

hi! I just replied to the issue you created. let me know if this helps <https://github.com/ploomber/projects/issues/24>

Thanks let me try and will share the example.

at this moment it's not possible to skip notebook generation. you could use functions to do that. this has come up a couple times and it'd be fairly easy to implement. can you open an issue?

thanks! Any other feedback? or something we can help with?

cool, don't hesitate to send any other questions!

let me see if I understood correctly, are you looking for something like this:
```git checkout v1 # go to tag v1
ploomber build # store outputs in products/v1
git checkout v2 # go to tag v2
ploomber build # store outputs in products/v2```


may I ask more a bit the use case? are v1 and v2 code versions that you want to keep active? in other words, having v2 does not replace v1

yes exactly,  the pipeline has some stages: get data, long post process, store to s3-parquet/postgres, (pstage) read them and "plot", (pp)produce byproducts.
the pstage and pp needs to be running several times, if someone finds problem need to fix and rerun, publish v1.1.  but for sure  vX.X are independent. they should be able to run in parallel.

can  ploomber  detect on which tag it is "sitting" and set appropriate variables? :thinking_face:

ok, yes it can! there's a feature that detects that but sadly it's undocumented. let me grab that info for you

it's funny because I implemented that a while ago for my projects but I never documented it :sweat_smile:

here's how you do it, add a prefix to your products:
```tasks:
    # tasks.get, features and join are python functions
  - source: tasks.get
    product: '{{products}}/get.parquet'

  - source: tasks.features
    product: '{{products}}/features.parquet'

  - source: tasks.join
    product: '{{products}}/join.parquet'

    # fit.py is a script
  - source: fit.py
    name: fit
    product:
        # that generates an html report as output
        nb: '{{products}}/nb.html'
        model: '{{products}}/model.pickle'

    # only show outputs (not code) in the report
    nbconvert_export_kwargs:
      exclude_input: True```

then create an `env.yaml` with this:
```_module: .
products: '{{git}}'```

in the same folder as the `pipeline.yaml`

to test it do `ploomber status`, you'll see that the prefix changes depending on the tag/commit

since it seems that you only want to version a few of the stages, you can apply the `{{product}}` prefix to only a few tasks, in that case some tasks can go to a common folder (say `products`) and others to the versioned folder (`v1/`, `v2/`)

ok need to go deeper next week and prepare simple use case + bundle it into docker+ci

sure, thanks for bringing this up! can you please open an issue? so I have a reminder to document this, and other people can see how this works in the docs

how to make products on the remote storage??

for uploading to remote storage you have 2 options. either you use the client directly in your script, or you use the built-in feature, here's an example: <https://github.com/ploomber/projects/tree/master/cookbook/file-client>

we're still lacking a bit on the remote storage documentation so if you have any questions, please send your questions

i am using minio, so mc client is my choice

alrigh thanks for opening the issue! since we don't have support for minio yet, you can use the python client directly. the on_finish hooks are useful here, you can register a function to execute when your task executes successfully, this is a good place to upload your outputs - docs: <https://docs.ploomber.io/en/latest/cookbook/hooks.html>

this is an interesting use case, please share your feedback once you start working on the implementation!

definitely will share, btw i was looking on the case with s3 examples, how should I define the bucket in the yaml file?

you define the bucket in the function that instantiates the client: <https://github.com/ploomber/projects/blob/f89e3f0a1095049f773c4a0a996c873d13332082/cookbook/file-client/clients.py#L10>

So kind of like an API reference guide?
The second link isn’t really python tasks but more of general concepts.
We do have this <https://docs.ploomber.io/en/latest/api/index.html|reference API>

yeah. I agree the link are general concepts, but there's a good point here. our documentation has no easy way for someone to quickly check how to assemble a shell, python, script task. So it's a good idea to add a new section to the cookbook with that info, then we can link from the user guide to the cookbook. so the idea is, if you want the full story -&gt; user guide, quick reference -&gt; cookbook. thanks for the feedback <@U029QTPAQ8H>, can you open an issue?

ok, can you edit the `pipeline.yaml` and change all `.html` files for `.ipynb`  ?

Executing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:02&lt;00:00,  1.41cell/s]
Executing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:02&lt;00:00,  1.93cell/s]
Executing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01&lt;00:00,  3.23cell/s]
Building task 'plot': 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:05&lt;00:00,  1.84s/it]
name    Ran?      Elapsed (s)    Percentage
------  ------  -------------  ------------
raw     True          2.15235       39.0844
clean   True          2.09616       38.0642
plot    True          1.25841       22.8514

thank you!!! that was just the example untouched so was confused why there was an error

you're good. it's an error in one of the dependencies that ploomber uses. you're using windows right? we've seen this error before

cool now I can get back to building :slightly_smiling_face:

nice, feel free to post any other questions you may have

was the error you posted the full text? cause i was expecting to see something `'charmap' codec can't encode character`

once I started getting the error I ran the example to see where the problem was

ok cool. i was worrying that ploomber might be hiding some of the error traceback, but if you are able to see the `'charmap' codec can't encode character`, then we're good

Error building task 'downloadData': the task ran successfully but the product is missing. Ensure the task is generating the declared products

ok so once I changed .html to .ipynb I get this

ok, so that means that in your `pipeline.yaml` your `downloadData`  task is supposed to generate some file, but after executing the code, ploomber cannot find it

only one of the two csv's in the product get completd

you need to ensure that your code is actually creating both files, can you share your `pipeline.yaml`?

tasks:
  - source: 00-data_collection/downloadData.R
    product:
      nb: 00-data_collection/downloadData.ipynb
      data: 00-data_collection/metadata/counterparts.csv
      data: 00-data_collection/metadata/countries.csv

yep, ensure is generating both counterparts.csv and countries.csv

dots.zip

so what's the problem here? it doesn't run?

yea it doesnt make counterparts.csv, though I ran the R code line by line and it works, and then error's out saying it can't find the file

oh i see what the error is. so you're saying in your pipeline.yaml that you'll save the output in the 00-data_collection folder but in your code, you're saving them in the current directory (the one that executes ploomber)

```# pipeline.yaml
tasks:
  - source: 00-data_collection/downloadData.R
    product:
      nb: 00-data_collection/downloadData.ipynb
      counterparts: 00-data_collection/metadata/counterparts.csv
      countries: 00-data_collection/metadata/countries.csv```

then in your script:
```
write.csv(countries, product$countries)
write.csv(counterpart, product$counterparts)```

no worries, this is very helpful feedback. ploomber should be easier to use :slightly_smiling_face:

if you want ploomber to extract the values from pipeline.yaml to your scripts, you can run `ploomber nb -i` (it does it when you run `ploomber build` automatically but with this command you can see how it'd look like)

ok I made the changes and getting this error

Exception encountered at "In [6]":
Error in if (file == "") file &lt;- stdout() else if (is.character(file)) {: argument is of length zero
Traceback:

1. write.csv(countries, product$countries)
2. eval.parent(Call)
3. eval(expr, p)
4. eval(expr, p)
5. write.table(countries, product$countries, col.names = NA, sep = ",",
 .     dec = ".", qmethod = "double")

# + tags=["parameters"]
upstream = NULL
product = NULL
# -

ploomber build
Executing:  86%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                     | 6/7 [00:02&lt;00:00,  2.35cell/s]
Building task 'downloadData': 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02&lt;00:00,  2.60s/it]

=========================================================================================== DAG build failed ===========================================================================================
------------------------------------- NotebookRunner: downloadData -&gt; MetaProduct({'data': File('00-data...ountries.csv'), 'nb': File('00-data...adData.ipynb')}) --------------------------------------
-------------------------------------------- C:\Users\yosty\Desktop\Desktop_Folder\14 - git\timeSeriesDOTS\ploomber\dots\00-data_collection\downloadData.R ---------------------------------------------
---------------------------------------------------------------------------
Exception encountered at "In [6]":
Error in if (file == "") file &lt;- stdout() else if (is.character(file)) {: argument is of length zero
Traceback:

1. write.csv(countries, product$countries)
2. eval.parent(Call)
3. eval(expr, p)
4. eval(expr, p)
5. write.table(countries, product$countries, col.names = NA, sep = ",",
 .     dec = ".", qmethod = "double")

ploomber.exceptions.TaskBuildError: Error when executing task 'downloadData'. Partially executed notebook available at C:\Users\yosty\Desktop\Desktop_Folder\14 - git\timeSeriesDOTS\ploomber\dots\00-data_collection\downloadData.ipynb
ploomber.exceptions.TaskBuildError: Error building task "downloadData"
=========================================================================================== Summary (1 task) ===========================================================================================
NotebookRunner: downloadData -&gt; MetaProduct({'data': File('00-data...ountries.csv'), 'nb': File('00-data...adData.ipynb')})
=========================================================================================== DAG build failed ===========================================================================================

# pipeline.yaml
tasks:
  - source: 00-data_collection/downloadData.R
    product:
      nb: 00-data_collection/downloadData.ipynb
      counterparts: 00-data_collection/metadata/counterparts.csv
      countries: 00-data_collection/metadata/countries.csv

ploomber build
Executing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:02&lt;00:00,  2.75cell/s]
Building task 'downloadData': 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:02&lt;00:00,  2.95s/it]
name          Ran?      Elapsed (s)    Percentage
------------  ------  -------------  ------------
downloadData  True          2.94519           100

yeah, my guess is that there was some error with the tags

great to see it working, what are you building?

I've been playing around with trade networks for fun <https://rcyost.github.io/xgboost-trade-forecast>

but its expanded into several data collect files and analysis files which is getting unmanageable

looking to string it all together with ploomber

unfortunately the dbnomics API offers metadata in R client and better data downloading in python client, so the downloading is in R +python haha

eventually I'd like this to be hosted on a cloud and have a front end, will look to leverage ploomber as much as possible!

sounds great! please keep us posted with progress and send any questions you may have!

it's up to you how to manage Python/R, as long as you have a working Python/R environment, it will work. yes, it works in docker

but how do I point it to a specific virtual environment?

activate the environment, then run `ploomber build`. is that what you mean?

yea this would work for python, and for R how would I specify the kernel?

idk if kernel is the right word but version of R

ah, good point. so ploomber will use the R installation that you registered when installing IRKernel. so if you switch to another R installation, you can install the IRKernel there

you can have more than one IRKernel installed at the same time and switch with one arg, hold on a sec

ok so if you have more than one IRKernel, you can list them with `jupyter kernelspec list` then choose one and pass it in your `pipeline.yaml`

```tasks:
  - source: script.R
    product ...
    kernelspec_name: kernel-name```

ok I was a little confused with the R example, I never saw iris being loaded

yes, this wasn't an active issue more of something I didn't understand but is versioning is a big pain so is a first Q i have haha

you can tell ploomber to expect a folder instead of individual files, then store your files there:
```tasks:
  - source: 00-data_collection/downloadData.R
    product:
      nb: 00-data_collection/downloadData.ipynb
      files: my-files/```

like I'm struggling on finding an example to copy for accessing tags in a .py file

maybe a walkthrough of one of these on the docs page <https://github.com/ploomber/projects/tree/master/templates>

thanks for your feedback. are you looking for an example to edit the cell tags? I' m not sure I'm following

```# %% tags=["parameters"]
upstream = ["raw"]
product = None

# %%
df = pd.read_csv(upstream['get']['data'])```


this was confusing me, how does the upstream reference 'get' if the only upstream is "raw"?

also it seems the advanced ML example totally ditches the .yaml file and builds the pipeline in python API, is this because the python API gives the user lower level or more functional control over the pipeline?

oh, that example seems wrong. is that somewhere in our docs?

yep, the ml advanced uses the Python API since it gives more flexibility, but it requires you to write more code

oh, I spotted the error. I'll fix it, thanks!

please keep this questions coming. it's pretty hard to write good docs, so all feedback is welcome, especially errors like this

would it be possible to number the steps on this page <https://docs.ploomber.io/en/latest/get-started/basic-concepts.html>

the page seems kinda hard to follow for complete noobs (for which the page is intended)

reference the .yaml in this diagram, it just kinda appears later in the page

why is the cell injection explanation in the Tasks: scripts/notebooks section and not in the others?

cell injection is key to all of ploomber no? I'd include it in a specific task section only if it was specific to this task?

to make this more clear, maybe show a .yaml of these task functions being used. its not clear how they get used in the overall orchestration from just this page

further there is a clean.py and a functions.clean

these are separate things but the common name in the docs makes it confusing

```tasks:
  # this is a sql script task
  - source: raw.sql
    product: [schema, name, table]
    # ...

  # this is a function task
  # "my_functions.clean" is equivalent to: from my_functions import clean
  - source: my_functions.clean
    product: output/clean.csv

  # this is a script task (notebooks work the same)
  - source: plot.py
    product:
      # scripts always generate a notebook (more on this in the next section)
      nb: output/plots.ipynb```


but in fact it is never used in the just referenced pipeline, kinda confusing?

i get you want to demonstrate each task primitive, but why no demonstrate the .py primitive with the later plot.py file?

I guess the ideal would be to demonstrate a minimal pipeline showcasing all 3 types of tasks and the required info to use each

not v easy, hopefully the ramblings of a total noob will help

this is great. thanks a lot. do you think it'd be better to have the same example implemented with scripts, functions and SQL?

what's your github handle? I'll tag you in an issue i'm writing now so you can provide more feedback there

that would be useful, but if it is more worth your time a single pipeline with 3 steps showing each of the 3 types of tasks a bit more clearly would be the baseline to get the basic info needed

i know :disappointed: we have an open issue to <https://github.com/ploomber/ploomber/issues/540|ditch it> - the best option is to use <https://docs.ploomber.io/en/latest/user-guide/faq_index.html?highlight=pygraphviz#plotting-a-pipeline|conda>

yea I don't use conda, I guess I'll have to wait for another option

Hi <@U034URH93QE>, I'm late here by 4 days. But can you try these steps mentioned <https://pygraphviz.github.io/documentation/stable/install.html#windows|Pygraphviz Windows Installation>? It worked for me. Please make sure you have vs studio cpp build tools installed already before executing commands mentioned in <https://pygraphviz.github.io/documentation/stable/install.html#manual-download|manual setup>

Assuming you're using ploomber + soopervisor with the argo backed. here are a few thoughts: the ploomber + argo works fine on GCP. now regarding the artifacts of the pipeline you have two options: you can configure cloud storage, this way ploomber will upload the outputs to the bucket and the next task will download them to use them as inputs. this is simple to setup but slows the pipeline execution a bit since each task needs to download its inputs before starting. the second option is to configure a shared disk so all pods share the filesystem. under this config, you don't need to upload/download artifacts. <https://soopervisor.readthedocs.io/en/latest/tutorials/kubernetes.html#google-cloud|our tutorial covers both use cases>. let me know if you have questions!

Thanks for the breakdown Eduardo, looking into the tutorials now. I'd probably look into option 2 to share a file system and then delete the intermediate artifacts after each run.

sure, let me know once you get that working, happy to help you with next steps to streamline the dev/deploy workflow :slightly_smiling_face:

and if you have questions with the tutorial, feel free to post another question on this channel

Hey Ido, Eduardo's method works great! We went with option 2. Ran Argo workflows with a shared attached disk to store the intermediate artefacts.

Running workflow pipelines is the first step, orchestrating it with proper resources is the next challenge. We're using these workflow pipelines to serve users so potentially, at spikes, for every 1 CPU/RAM resource we have, we might have workflows that require 3-5 CPU/RAM resources total. To serve all of the workflow requests, we'll need an intelligent queue control system.

The current challenge we are figuring out is that Argo workflow has no queues. This causes two problems for us. Firstly, if the pod's resources are fully utilized, the incoming workflows cannot be accepted. Secondly, assuming we use some form of queue system, if the pod's resources are partially utilised but are unable to fit bigger workflows, we haven't found an "intelligent broker system" that is able to fit smaller workflows (smaller datasets = smaller workflows) that are in the queue.

E.g.
Queue = 1, 2, 3, 4, 5
Pod 1 = 3/5 resources used -&gt; I could fit in 1 / 2 here but not 3, 4, 5
Pod 2 = 2/5 resources used  -&gt; I could fit in 1 / 2 / 3 here but not 4, 5

Would you suggest any approach?
<@U037XUTKBBK> for your reference

did some quick research. yeah, argo doesn't have a notion of a queue so based on <https://stackoverflow.com/questions/63956383/creating-a-queue-system-with-argo-workflows|this>, looks like you need to check the resources before submitting the workflow - check out the answer it provides a few interesting points. I'm no k8s expert but I'm guessing there's a way to to know how much resources are available, then use that to determine if accept reject a workflow submissions?

Okay gotcha, I'll look into this. Yes, knowing how much resources should be 100% doable. The tricky part is knowing the resources required for every workflow in the queue such that we can intelligently fit smaller workflows into a pod on nodes with spare resources.