No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

Hello community, I encountered a confusion while going through the documentation of the project...
In the given pic, I think the `upstream = ['clean']`  will make more sense as plot has clean as an upstream and plot can't be an upstream as its the last task of the pipeline (as per the example)

It is one of their default examples. Personally I found kubeflow crazy complicated and hard to port into Slurm, but I think they might be the only framework which supports this kind of looping.

During the execution the pipeline "unfolds" into a dag.

looks like k3s and k3d are pretty similar

Ok now I am running into a problem when trying to re run the pipeline

```Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/ploomber/tasks/abc.py", line 563, in _build
    res = self._run()
  File "/opt/conda/lib/python3.9/site-packages/ploomber/tasks/abc.py", line 647, in _run
    raise TaskBuildError('Attempted to run task "{}", whose '
ploomber.exceptions.TaskBuildError: Attempted to run task "get", whose status TaskStatus.Skipped. Render again and set force=True if you want to force execution
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/ploomber/cli/io.py", line 34, in wrapper
    fn(**kwargs)
  File "/opt/conda/lib/python3.9/site-packages/ploomber/cli/task.py", line 56, in main
    task.build(force=args.force)
  File "/opt/conda/lib/python3.9/site-packages/ploomber/tasks/abc.py", line 531, in build
    res, _ = self._build(catch_exceptions=catch_exceptions)
  File "/opt/conda/lib/python3.9/site-packages/ploomber/tasks/abc.py", line 582, in _build
    raise TaskBuildError(msg) from e
ploomber.exceptions.TaskBuildError: Error building task "get"```


ahaaaa, sorry now I understood what kind of Dockerfile are you asking for. The one from the training folder. I was trying to reproduce the whole sequence inside the docker image :smile:

This is the Dockerfile from training folder. ml-intermediate example. It fails on:
```soopervisor export training --skip-tests --mode force```
with the following error:
```FileNotFoundError: [Errno 2] No such file or directory: 'dist/ml-intermediate.tar.gz'```
Thanks for patience. :slightly_smiling_face:

this is the whole sequence:
```
2432  ploomber examples -n templates/ml-intermediate -o ml-intermediate
 2433  cd ml-intermediate/
 2434  ploomber install
 2439  conda activate ml-intermediate
 2440  soopervisor add training --backend argo-workflows
 2441  vim soopervisor.yaml 
 2442  soopervisor export training --skip-tests```
throws the above mentioned error.

here is also the soopervisor.yaml and Dockerfile from the sequence above.:

Thanks <@U0164GZJ5EK>, now it works perfectly. I am able to run projects on both Slurm and Kubeflow.

What would you suggest would be the best way to add volume into the pipeline that would work on Kubeflow cluster?

So I have tried kubeflow to execute the argo.yaml of ml-advanced example. Interestingly it seems to have cashed some of the steps from ml-basic and it is reusing the output.

It is actually causing some errors, it seems that when argo task has the same name and the same input the kubeflow will used cached output. I am not a kubeflow expert tho.
```
time="2022-01-18T16:24:47.259Z" level=info msg="capturing logs" argo=true

  0%|          | 0/5 [00:00&lt;?, ?it/s]
100%|██████████| 5/5 [00:00&lt;00:00, 5318.67it/s]
Traceback (most recent call last):
  File "/opt/conda/lib/python3.9/site-packages/ploomber/cli/io.py", line 34, in wrapper
    fn(**kwargs)
  File "/opt/conda/lib/python3.9/site-packages/ploomber/cli/task.py", line 56, in main
    task.build(force=args.force)
  File "/opt/conda/lib/python3.9/site-packages/ploomber/tasks/abc.py", line 518, in build
    raise TaskBuildError(
ploomber.exceptions.TaskBuildError: Cannot build task 'sepal-area' because the following upstream dependencies are missing: ['get']. Execute upstream tasks first. If upstream tasks generate File(s) and you configured a File.client, you may also upload up-to-date copies to remote storage and they will be automatically downloaded
Error: exit status 1```


Somehow the ml-intermediate  argo.yaml pipeline runs correctly but the output is not written onto the shared-folder when executed in cluster.

What might I be doing wrong?

thanks for your patience with my questions.

I dont think that the pipeline is not being executed. It actually prints the progress bars in logs and ebverything in the final component.

I will try what you suggested.

```cat env.yaml
sample: False
root: /mnt/shared-folder```
```(base) root@4c4b0d4a1213:/project# cat pipeline.yaml
meta:
  jupyter_functions_as_notebooks: True
  import_tasks_from: partial.features.yaml

#clients:
#  File: clients.get_s3

tasks:
  - source: tasks.get.fn
    name: get
    product: '{{root}}/sample={{sample}}/get.parquet'
    params:
      sample: '{{sample}}'

  - source: fit.py
    name: fit
    product:
        nb: '{{root}}/sample={{sample}}/nb.html'
        model: '{{root}}/sample={{sample}}/model.pickle'```
ploomber status is attached:

(ploomber build runs correctly)

*Keep it mind I am running the argo.yaml in Kubeflow.* 

See added picture, everything works as it should

However Kubeflow itself does not show the volume as mounted for the pipeline steps.

As I mentioned, the only way for me to view the contents of the pvc is through the notebook server. (At least it seems.)

And there the volume seems empty. Though it might be that my current user does not have the correct permissions.

Hey <@U056DDC3H4N> welcome! In which stage do you see it happens? Are you using your own pipeline or the sample one?

I just ran the sample pipeline, plot-with-raw-cell and it seems to pass so I’m assuming this happens with a custom pipeline. Did you try running it locally and it passes? (It looks like you’re trying to pass an argument that was deprecated)

These are the last rows of the docker image logs.

I simply downloaded the input-data.ipynb and the penguins.csv files through the links indicated in the same tutorial.

And then I run them on Plomber Cloud site.

Interesting, let me dive into this.
Did you upload both the csv and the notebook?

I was able to recreate on the cloud, researching. (as an FYI it’s probably some parameter/function call in the notebook that’s deprecated).

<@U056DDC3H4N> Please take a look <https://github.com/ploomber/projects/blob/master/guides/cloud-notebooks-user-guide/README.ipynb|on this guide> (Similar concept, more detailed and a different notebook). The API itself seems to work fine, there’s something with that notebook that we need to debug, I’ve opened an issue <https://github.com/ploomber/projects/issues/85|about this>, you can comment to get updates. Feel free to raise more questions and thanks for raising the issue!

Although I followed this tutorial I got the same error.

I assume that it broke when passing an input, since when I tried a plain one it passed.

try connecting by creating an engine and then passing the engine object, let me know if that helps <https://jupysql.ploomber.io/en/latest/connecting.html#using-an-existing-sqlalchemy-engine-engine>

This message contains interactive elements.

Great! Can you open an issue? I think we should update the docs so we recommend users connect this way 

If I alias the connection I can than close it out using the alias.