No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

Current ones are small (&lt;100G, 50k samples), but I also want to make it work with ones that are 10-100T in size.

But even with the small datasets, I would love to have them processed in parallel. 

Ploomber is more like an orchestrator than a distributed computing solution, but if you're doing simple stuff like mapping, you can implement it with the grid feature: <https://docs.ploomber.io/en/latest/cookbook/grid.html>

essentially, you could define a grid of parameters and each of those parameters could be the indexes to process e.g., first task processes the first 1 M rows, the next one another 1M and so on. parquet is a a great format for these type of things

however, for more complex operations, I'd recommend you go with a distributed computing framework

Thanks, this makes sense. Maybe I could make this type of processing a single step in the ploomber pipeline.

yeah that might work. feel free to ask any other questions!

hey <@U04BHMBE9GR>! if your task generates more than one file you can declare a folder as the product:

```tasks:
  - source: tasks.create_files
    product: path/to/folder```
then store the files in that folder. does this solve the issue?

no problem! feel free to <#C028KJCJDRT|>!

so, the down-strream tasks will get the path dir, and they should glob this file and read all those files?

yes. you can glob them and read some or all of them, depending on what  you want to do

looks like an issue with the sqlite version you're running. can you share your version?

```import sqlite3; sqlite3.sqlite_version```

```import sqlite3
sqlite3.version
'2.6.0'```
Strange because my system claims I have 3.7 :shrug: will figure out how to update

yeah. that's strange. I'd assume the version would start with 3.x.y

np, let us know if we can help with anything!

heyo, after building some custom pipelining functionality on mlflow, I started looking for better software that can beat what I created - with a similar amount of effort. problem has been that up till now, all I found could make things way better but also require way more effort. thats why im here - ploomber seems to meet my requirements well. the caveat is though is that in my team we want to align our data science development with our software development, meaning that any type of pipeline is a child class of the more general pipeline parent, which has list as an attribute with the methods that have to be executed - inheritance makes it easier to create new pipelines that are related

the docs say you can use scripts, nbs, or funcs as tasks. what isnt clear to me is if class methods would qualify as funcs or would it break

It sounds like using the Python API might work better for you since you'll be able to write a wrapper that adapts to the class-based workflows: <https://github.com/ploomber/projects/tree/master/python-api-examples>

pipeline:
```tasks:

  - source: tasks/raw.py
    product: ./test_products

  - source: tasks/get.py
    product: ""```


get.py
```# + tags=["parameters"]
# declare a list tasks whose products you want to use as inputs
upstream = None

# -

import glob
import pandas as pd



# +
upstream = ['raw']
product = None
# -


print('hello')```


raw.py
```import pandas as pd
import numpy as np

upstream = None
product = None

for i in range(10):
    data = np.random.random((10,10))
    pd.DataFrame(data).to_csv(f'./test_products/rand_{i}.csv')```

test_products is the path to the directory

this isn't officially supported but this should work.

start a console with `ploomber interact`, then

 `dag['task_name']._post_run_actions()`
then verify the new status with `ploomber status`

post_run_actions will run the on_finish hook (if any) and then overwrite the metadata without running the task. if this works for you. please post this on the github issue in case anyone else needs this recipe

did `_post_run_actions` fix your issue <@U02HEAPSDHR>?

I didn't check. I went ahead and just ran the model and did something else. But I will - maybe next week? Good luck with the presentations this week at PyData <@U0164GZJ5EK>!

Finally needed to do this and it worked! That's super helpful. Thanks <@U0164GZJ5EK>

is "CAPSL-NASUNI" a product of the data_pull task? based on your pipeline.yaml it looks like it. but based on the error message, it looks like you're trying to read it (so it might be an input?)

Many of our servers begin with the name CAPSL but I am not specifying this in my data_pull task so I am not sure how Ploomber was able to detect it

so it sounds like this is an input data file right? in such case, it should not be part of the `product` section

data_pull's job is to write a csv file to the I drive

can you share the full traceback? I'm a bit confused about the error

Loading pipeline...
Executing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:49&lt;00:00, 6.14s/cell]
Executing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:09&lt;00:00, 1.23cell/s]
Executing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:12&lt;00:00, 1.26cell/s]
Executing: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:09&lt;00:00, 1.20cell/s]
Executing: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [01:51&lt;00:00, 14.00s/cell]
Building task 'data_table_construction': 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [03:13&lt;00:00, 38.64s/it]

looks like the pipeline is running without errors? I don't see any traceback

that is the weird part, the pipeline finishes, but then I get

```============================================================================================== DAG build failed ===============================================================================================
----------------------------------- NotebookRunner: pull_weekly_tcd_raw_data -&gt; MetaProduct({'data': File('\\\\CAP...\\vessel.csv'), 'nb': File('product...w_data.ipynb')}) ------------------------------------
-------------------------- C:\Users\lfunderburk\OneDrive - Seaspan Corporation\Documents\GitHub\maritime_technology_weekly_tcd\weeklytcd\pipeline\scripts\pull_weekly_tcd_raw_data.py --------------------------
Traceback (most recent call last):
  File "C:\Users\lfunderburk\Anaconda3\envs\venv\lib\site-packages\ploomber\tasks\abc.py", line 583, in _build
    self._post_run_actions()
  File "C:\Users\lfunderburk\Anaconda3\envs\venv\lib\site-packages\ploomber\tasks\abc.py", line 357, in _post_run_actions
    self.product.metadata.update(
  File "C:\Users\lfunderburk\Anaconda3\envs\venv\lib\site-packages\ploomber\products\metadata.py", line 359, in update
    p.metadata.update(source_code, params)
  File "C:\Users\lfunderburk\Anaconda3\envs\venv\lib\site-packages\ploomber\products\metadata.py", line 263, in update
    self._product.save_metadata(data)
  File "C:\Users\lfunderburk\Anaconda3\envs\venv\lib\site-packages\ploomber\products\file.py", line 74, in save_metadata
    self._path_to_metadata.write_text(json.dumps(metadata))
  File "C:\Users\lfunderburk\Anaconda3\envs\venv\lib\pathlib.py", line 1255, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors) as f:
  File "C:\Users\lfunderburk\Anaconda3\envs\venv\lib\pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "C:\Users\lfunderburk\Anaconda3\envs\venv\lib\pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '\\\\CAPSL-NASUNI\\path-to-files\.vessel.csv.metadata'```

this happens for every step in the pipeline

ok, this looks like a bug. is this the actual value that you have in your pipeline.yaml? `data: I:/path-to-files/file_name.csv`? I can dig into it, but as a workaround, you can change the products to have relative paths and move them with another script to the `I:` drive

Thank you Eduardo. Not posting the full path for security reasons

not necessary. I'm guessing it has the `I:/something/another.extension`  format, right?

I ran into a permissions issue when I tried it.

Likely due to security configuration of my company. Will try a couple more things and inquire internally.

`tasks:`
    `- source: scripts/data_pull.py`
      `product:`
          `nb: products/data_pull.ipynb`
          `data: I:/path-to-files/file_name.csv`
`...`

The pipeline above assumes that data_pull.py generates and saves a file_name.csv in the specified I drive
This I drive is under company network

Where is ploomber getting CAPSL-NASUN from? why can't it access it even though the script successfully generates and saves a csv file in the location specified?

sure, happy to help!
&gt;  ml-online conda environment is not getting created by default
can you point me to the command where you'd expect the conda env to be created?

are you trying to run that example or are you using it as a guide for your project!

```# configure development environment
ploomber install

# activate environment
conda activate ml-online```
I get an error that the environment ml-online does not exist. I am running the example as it is.

Ok, run `conda` commands instead of `ploomber install` , and see if you can activate the environment: `conda env create -f environment.yml`

The environment gets created. However, the dependencies are not exported to the lockfile due to conda versions `conda env export --no-build --file environment.lock.yml`

`InvalidVersionSpec: Invalid version '!=~5.0': invalid operator`

I saved the dependencies in a requirememts.lock.txt via pip freeze.

This is where the next problem comes with soopervisor export. I get `No module named 'ml_online'` . This occurs when trying out both templates ml-online and ml-intermediate.

yeah, try using requirements instead of environment.yml. ml-intermediate should not require ml_online, try starting from scratch. there might be something going on with your environment.

also, I'd recommend you trying these instructions: <https://ploomber.io/blog/ds-platform-part-iii/> - this tutorial is more recent

the 3rd part has the commands you need to submit the jobs to AWS Batch, but you can look at the other parts, they cover setting up the infra

Yeah. You can use the —force parameter and it will run all steps in the pipeline 

Thanks for reaching out! We're looking into it, thanks for letting us know! Cc. <@U0164GZJ5EK> &amp; <@U025MBQ6W7P>

This is fixed now :slightly_smiling_face:

You can set an upstream depending on the actual values. Check <https://docs.ploomber.io/en/latest/api/spec.html#tasks-grid|this one out>. Let me know if that helps?

Yeah, I wasn't sure if "task-*" would work. That basically solves it. Thanks!

Good question! We’ve started with supervised classifier and regressors. Now we added some clustering plots like silhouettes and elbow curve. Is there something else you’re interested in? Feel free to open an issue and we can prioritize it

Traceback:
```Traceback (most recent call last):
  File "path-to-package-repo/venv/lib/python3.9/site-packages/ploomber/cli/io.py", line 103, in wrapper
    fn(**kwargs)
  File "path-to-package-repo/venv/lib/python3.9/site-packages/ploomber_core/telemetry/telemetry.py", line 545, in wrapper
    raise e
  File "path-to-package-repo/venv/lib/python3.9/site-packages/ploomber_core/telemetry/telemetry.py", line 533, in wrapper
    result = func(*args, **kwargs)
  File "path-to-package-repo/venv/lib/python3.9/site-packages/ploomber/cli/nb.py", line 472, in main
    params_to_inject, warning_message, exceptions = _get_params_to_inject(
  File "path-to-package-repo/venv/lib/python3.9/site-packages/ploomber/cli/nb.py", line 132, in _get_params_to_inject
    unprioritized_tasks_to_inject, warning_message = _get_tasks_to_inject(
  File "path-to-package-repo/venv/lib/python3.9/site-packages/ploomber/cli/nb.py", line 84, in _get_tasks_to_inject
    full_template_name = Path(task.source._path).name
AttributeError: 'PythonCallableSource' object has no attribute '_path'```

this seems like a bug, looks like your pipeline has some Python function tasks and ploomber is treating them as scripts/notebooks. can you open an issue on github? as a workaround, you can remove the Python function tasks in your pipeline.yaml, run the command and then add them back again

Thanks Eduardo. I posted an issue. Appreciate the suggestion for a workaround.

Do you mean locally? check out the Jupyter API reference, it seems this is false by default, you can specify a config.json to load on start and set it to true:
<https://jupyter-server.readthedocs.io/en/latest/other/full-config.html>

Yeah I looked at those docs for a while but it was just under `~/.jupyter` and I wanted something under `project/.jupyter` but I also thought the need to have `--config $PWD/.jupyter/jupyter_lab_config.py` on the launch command was a foot gun

I found it okish with code stuff, it doesn’t debug well or give the optimal solution usually. Sounds like it’s a good Stackoverflow substitute though :smile:

btw, <https://ploomber.io/blog/cloud-notebooks-gui/|check out our new GUI>, should be pretty helpful with notebooks!

Yeah, I was avoiding playing with it for some reason until I heard it was just a good search engine replacement.

For any poor souls coming after me, that didn't work :disappointed:

hi! yes! ploomber is agnostic. you can use any library you want inside your pipeline. if you have any issues or questions, feel free to ask!

the problem solved, it the notebook does not print; but the terminal does it well

how are you running your pipeline? are you using <https://github.com/ploomber/soopervisor/|soopervisor>?

Hi Eduardo, I'm just using ploomber build, for the specific case of the task, I'm using ploomber task SOMETASK -some-envs--bla BLA

if by "requirements" you mean "virtual envs", then it's possible if your tasks are scripts or notebooks. You can pass `papermill_params` to your task and pass a different `kernel_name` . of course, this involves some setup since you need to ensure you have multiple kernels registered and they are discoverable by your current environment. is this what you need?

what I'm not following clearly is how you're planning to distribute in different computers

Thank you for your answer. What I mean is the following:  Assume a pipeline with certain tasks. Some tasks process (transforms) data, specifically, data cleansing and geospatial operations. After a task is executed the output is written in a standard format that will be parsed by the next task. Now, there is step that requires intensive computational power and, therefore, I want to run this task in a high performance computer. As this computer has limited resources, I don't want to install the entire environment that I use for the entire pipeline. I only want to install the packages that are necessary and sufficient for the specified task. I have seen that ploomber has a mechanism for installing software ,through virtual environments, using a requirements.yml file (I suppose). and then doing somehing like conda env create -f requirements.yml (for example in the case of conda). My question is: Does this feature is exclusive for the entire pypeline or, if it's possible to make ploomber to compile a requirements.yml file for a specific task? or, more likely, I am misunderstanding the feature of ploomber install ...    Thank you for your time and your patience.

thanks for the explanation! the `ploomber install` command installs packages for the whole pipeline. to solve your problem you'd have to manually list the dependencies that the task you want to execute requires and create the requirements.txt, then execute it in a different machine.

In Ploomber Cloud, we actually have this functionality, and we're able to infer a requirements.txt based on the contents of a jupyter notebook. most of the logic is open-source so if you wanna take a look, <https://github.com/ploomber/k2s/blob/58ae7efd233b4310dae940adc1bb20ffba7a777e/src/k2s/parse.py#L327|it's here>. but for your use case, it's probably simpler to just create the requirements.txt file manually

ok! thanks, I will take a look. I though that that functionality may be available because of the general dependence inference that ploomber is doing for the whole pipeline. But I see where are you going with ploomber Cloud. Thanks for the explanation. It is a really neat project! Thank you for your time and work.

sure! feel free to ask any other questions!

you can pass dictionaries as keys in grid, but I'm unsure if this is what you're looking for:

```grid:
  some_dict:
     some_key: some_value```
then, you can refer to the values in the <https://docs.ploomber.io/en/latest/cookbook/grid.html|task names>:

```name: some-task-[[some_dict.some_key]]```
Note: We have not tested the dot notation (`[[something.another]]` ) since we never thought about that use case but my guess is that it should work since we're just passing jinja variables to render the task name. so if you have issues let me know!

Yeah I mean I'd rather have those configs somewhere else than env.yaml or pipeline.yaml because I have many of them

For now I just pass their filenames as params in grid - how does Ploomber compare to Hydra? :smile:

I mean Hydra kinda does this, but I'd much rather have small subset of Hydra functionality in Ploomber than move from Ploomber

For starters I remember I tried using decorators to add type validation (I mean I'd load JSON from somewhere and then convert it to my config classes) and it was cumbersome

A concrete idea for implementation would be to add integration with dataclasses, something like Pydantic in FastAPI

<@U0164GZJ5EK> thanks for being a rubber duck XD I think I'll write a proposal for this (I imagine it won't be hard but it requires understanding how ploomber calls stuff)

it's a bit difficult for me to understand how the API looks like since i've never used Hydra, so yeah, feel free to write an API proposal, we can work it out and then you can submit a PR. I can help navigating the different modules to understand how ploomber calls things!

Hi. When pushing to my fork of Soorgeon, test are failing(see msg below). The weird thing is that, I even pushed a copy of the main branch (which had previously passes all tests) &amp; now that's failing too. I have researched but can't find an answer or figure out the problem.

hi! thanks for joining our community! we have a section in our docs with generic instructions: <https://jupysql.readthedocs.io/en/latest/connecting.html>, we use SQLAlchemy under the hood so you can check the docs as well: <https://docs.sqlalchemy.org/en/20/dialects/mysql.html>

we're currently working on add specific examples for each major engine to simplify this!

I tried this and got

%sql <mysql+pymysql://userid:pwd@hostname/dbname>

```Connection info needed in SQLAlchemy format, example:
               <postgresql://username:password@hostname/dbname>
               or an existing connection: dict_keys(['duckdb://'])
No module named 'pymysql'
Connection info needed in SQLAlchemy format, example:
               <postgresql://username:password@hostname/dbname>
               or an existing connection: dict_keys(['duckdb://'])```


looks like you’re missing the pymysql package 

Thanks, so silly of me to forget the basics.  Got so carried away with the SQL.

It works now.

No worries! To be fair, the error message is kind of confusing, we’re working on improving these parts, so if you have any other feedback please let us know!

Please join <#C03R6FJ7H2R|> as well :slightly_smiling_face:

How does ploomber know that a task has been executed? It is not solely based on the existence of the product files, right? I'm having problems running a single task without running the entire pipeline.

How can I override an environmental variable with nb --inject. For example: `ploomber nb --inject --env--VAR1 newvalue` ?

ploomber stores `.metadata`, they're prefixed with a dot so they're hidden by default, but you can see them with `ls -la` . that stores the task's last execution timestamp and source code. the rule to determine execution is either an upstream dependency has a higher timestamp or the source code has changed

you can run single tasks: `ploomber task {task-name}` . does that work? if you want to ignore status, you can force it with the `--force`  flag.

currently, there is no way to pass custom values to `--inject` so yeah you have to modify the env

Yes, I was using ploomber task but not with the --force flag!  I think it will do what I want. Thank you Eduardo! Ploomber is great!

awesome! feel free to ask any other questions

<@U03UR6Z9D27> are you using only ploomber or any other package? (`jupysql, sklearn-evaluation, ploomber-engine` )

The version has nothing to do with it, it exists since day one.

it's a leading dot with .metadata extension

but they are created after you run a task

so, I'm doing ls in the same level as pipeline.yaml, the only hidden files are: .gitignore and .ipynb_checkpoints

ah sorry, forgot to mention. they're stored next to your products

so if you store `products/stuff.csv` , it'll be `products/.stuff.metadata`

ooh, I get it now. Yes, I found them! Ok, now the problem is that the flag --force seems not to work

ahh, I know what is happening, Some part of the pipeline were run in a cluster. I only copied the products, not the metadata

yep, you need both so ploomber can recognize them

I'll add the metadata to the other piece of the pipeline and see what happens,  Thanks!

in your pipeline.yaml, set

```meta:
  extract_upstream: false```
then you'll be able to pass `upstream` to your tasks. but keep in mind that you'll have to do it for all tasks

I have another question, when I build a pipeline it starts but it stacked always from the beginning
Executing:   3%
do you have any suggestion to solve it

try adding some logging, might help you understand where it's getting stuck: <https://docs.ploomber.io/en/latest/user-guide/logging.html>

Hello… I keep getting: @deprecated.method(deprecated_in=‘0.21’, removed_in=‘0.22’)
TypeError: method() got an unexpected keyword argument ‘removed_in’