No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

<https://docs.ploomber.io/en/latest/user-guide/editors.html#using-git-hooks>

Oh lol, just noticed the “doesn't work on Windows” part :rolling_on_the_floor_laughing: 

Still good to know that this is the direction of the solution

<@U032HTM8DBN>: I remember implementing the git hook and seeing some errors on github actions on windows but not locally. you might still wanna give it a try. if you encounter issues, let us know!

Ok great — I’ll do that and report back either way

I see what the errors are. Git is using Unix the path separator to invoke the hooks (resulting in path not found) and anyway the hooks are bash scripts. It seems I may be able to use WSL (Windows Subsystem for Linux) to overcome this. If so I’ll write it up and submit a docs PR

I finally got around to trying this on Linux. One issue I’m having is that my repo has multiple projects, each with its own virtual env. So the virtual env is not at the same level as the .git dir. When I do the hook injection, ploomber wants me to run that from the top level. But when I commit, the hook can’t find ploomber because it’s a level in

I can just manually remove the injected params with the —remove option but thought I’d mention this in case others run into it

yep <https://github.com/ploomber/ploomber/issues/832>

thanks for your feedback! we'll add this to our backlog :slightly_smiling_face:

yes. Still lots of pain, unfortunately :disappointed: but if you're going Mac, then I'd still opt for M1/M2

I have a Mac Mini with M1 at home. At the beginning I faced some difficulties with some packages, since not all work with the architecture. Fortunately, a time has passed and there are fewer issues now (but they still exist!).

In my case, I'd still opt for M1/M2 too.

we've been overwhelmed with work :sweat_smile: sorry about the delay, most likely, we'll release it during the 2nd or 3rd week of october

I think ploomber has the potential for getting selected as an GSOC organization. Why don't we try getting it selected as an organization in upcoming GSOC event. <@U0164GZJ5EK>.

Is this about the security or the Google event?

Hi Joe, if you're planning to use Ploomber for a well structured project, we have a repo with some guides and examples that might help: <https://github.com/ploomber/projects>

You may need to add your data, create your own models, etc., but those examples can be a good start.

Good luck!

this one covers training and deployment as an API. <https://github.com/ploomber/projects/tree/master/templates/ml-online> but feel free to choose any example you like. and feel free to ask any questions!

I tried to to implement the templates but I will be honest I wound up completely lost. This is so different than what we are taught in class, and I am so new to this that I am not able to navigate this successfully.  I do not think i can use it for a project due to my time limitations but I would like to to learn how to.

Interesting, even the <https://binder.ploomber.io/v2/gh/ploomber/binder-env/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252Fploomber%252Fprojects%26urlpath%3Dlab%252Ftree%252Fprojects%252Fguides/intro-to-ploomber%252FREADME.ipynb%26branch%3Dmaster|initial example>?

correct. you wanna do some time series forecasting (<https://www.analyticsvidhya.com/blog/2021/07/time-series-forecasting-complete-tutorial-part-1/|check this out>), because that will allow you to incorporate past data to predict a future observation.

We went through YC so we launched on hacker news and appeared on the front page. I think that was in February 

Ahhhh that would explain it.  Very cool thanks.

Lots of very impressive growth even before that - congrats.

I see it : <https://news.ycombinator.com/item?id=30195191>

Hey hey, I know there were a lot of people here are researching chemistry and bioinformatics, any specialists in the area we can talk to? alphafold/colabfold/rna-seq/crispresso/sars cov-2? :smile:

absolutely! helping with docs is a great way to start. we have a guide to help people get started with contributing: <https://github.com/ploomber/contributing>

feel free to <#C028KJCJDRT|> if you have questions!

do you mean the template task? Currently no, we added it to facilitate the concept explanation for new users

<@U039FB8GYRL> has been doing a lot of stuff with R. :grinning: but I'm unsure if she has experience mixing R and Python.

I also did a bit of this successfully, I was using parquet to allow R/Python interoperability. since you can install the R interpreter and R packages with conda-forge, you can declare a single environment.yml to document all your dependencies.

please share your experience with us!

Hi! I have a project that switches seamlessly between my python scripts and my r scripts using ploomber and conda as the “glue”. The main quirk we experienced were needing to _initialize_ our conda environment with both R and Python. We’ve also had some conda troubles when switching machines, but we think that was a conda problem not a ploomber problem.

We havent tried e.g. using magic cells in a jupyter notebook to switch between R and Python in the same jupyter nb but we may need to in the future when we start generating our plots + switching to ggplot in the same analysis

thanks for sharing your experience! for solving thr problem when switching machines. have you tried `conda env export --no-build &gt; environment.yml`? this will produce a portable yml file that you can use to re-create environments on other machines

<@U039FB8GYRL> is there a use case of running both R and python in the same notebook with magics?

in my very humble opinion, not really lol but some of my collaborators like to do exploratory modeling and on-the-fly plotting in the same notebook

we also heavily use some r packages/api for some of our datasets (shameless plug for <https://github.com/langcog/childesr> and <https://github.com/langcog/wordbankr>), though im working on porting some over to python (even more shameless plug for <https://github.com/langcog/childespy>) so its nice to pull the data in R, do our more ML adjacent stuff in python, and then report the results in R

So you have a separate environment.yml for conda. What d u use for R environment management and how does it blend together in the env.yml of conda? I.e. How does your project set up look like? 
I mean the following:
`Conda env create ... 
Pip install -e` 

yeah we have a .yml for our conda environment that has both r and python like…
```name: proj_pipeline
channels:
  - conda-forge
  - defaults
dependencies:
  - ploomber=0.19.4
  - python=3.8.13
  - r-base=4.0.5```

If u have specific r libraries where d u put them? Just r-.... ?

for most of them! we have some that arent on forge. in that case, our full setup commands are
```conda env create -f proj_pipeline.yml

conda activate proj_pipeline
ploomber scaffold --entry-point pipeline.preprocessing.yaml
ploomber scaffold --entry-point pipeline.modeling.yaml
#install childesr in R, not conda
R
install.packages(c("childesr", "dclone"))  # mirror 76

#setup the nltk
python
import nltk
nltk.download('wordnet')
nltk.download('omw-1.4')
nltk.download('semcor')```

And can you conda export the complete .yml including the specific packages or r?

so we enter R from within our conda environment and install the non-conda R packages manually

I see. Not the neatest solution but it works.

I dont think so, i think conda export skips over the non-conda packages?

either way theyd fail to install because theyd search on forge and not find them

we’ve considered doing some work for _our_ R packages and adding them to conda forge, but we cant do anything for other folks (like dclone)

I LL have to try and play around with this. Perhaps, R has very different idea on how reproducible environments should be done. I wonder if this mixing is a good idea or just trouble waiting to happen. For me even just pure conda often fails to keep reproducible env.

yeah there are some reproducibility focused R packages, but they all function within R and dont play nice with conda

e.g. <https://rstudio.github.io/renv/articles/renv.html>

so u do not mix conda + renv for the underlying R

when running a task, ploomber stores the source code and parameters. if you try running the task again but the code and parameters have not changed, it will skip it. we store the code and parameters in a metadata file that has JSON format. this means that there's only certain data types that we can store (only the ones that we can store in a JSON file). this warning is telling you that changes to the "dag" parameter wont be detected. so if you run the task, and change the parameter, ploomber still thinks the task is up-to-date. here's how to supress it: <https://stackoverflow.com/a/56142903/709975>

looks like you're passing a task as parameter to another task, what's the use case?

Does this happen with multiple tasks you’re trying to parallelize or a single one?
Ideally each module should have its own task, that way you can reuse it and enjoy the parallelization.

I think Jan wants to parallelize inside a task.

I'm guessing you're using functions as tasks. By default, ploomber runs them in a subprocess to ensure memory is cleared up but you can change this setting: <https://docs.ploomber.io/en/latest/api/_modules/executors/ploomber.executors.Serial.html#ploomber-executors-serial>

```executor:
  dotted_path: ploomber.executors.Serial
  build_in_subprocess: false```

hey <@U043N03SEKH>! you're on the right track `params.resources_` is the way to go currently. however, since `params.resources_` looks at the whole file you might get some false positives (e.g., task "some-task" calls function A in utils.py, but since `resources_` looks at the whole file, if you modify function B, "some-task" will execute again)

we have a stalled <https://github.com/ploomber/ploomber/tree/dev/sourcetree|branch> that will provide a much better experience here and will automatically detect deep dependencies. e.g. if "some-task" calls function A in utils.py, only chances to such function will cause the task to re-run. the implementation is almost complete but it needs more testing, so if you're up for the challenge, feel free to take on it!

That's great - I'll take a look :slightly_smiling_face:

this error happens when ploomber is unable to find the inputs to execute the task you requested. so two things might be happening: either the input files (products from the upstream task) are not there or the files are there but have different names.

can you download the docker image locally? that'd be the best way to debug. fetch the image, start an interactive session and try to run: `ploomber task {task-name}` , once you see the same error. see if the upstream products exist

`but have different names.` what would be the reason for that. I see the files in the S3 bucket so the previous task successfully wrote them. IT does run fine. The 3 parallel child tasks all error with the same message that they cant find the deps. Also they dont mention a specific file but the task as a whole `dependencies are missing: ['data'].`
```  - source: scripts/data.py
    name: data
    product:```

ah, I think i know why! The `{{now}}` placeholder is evaluated when the task starts execution. so it'll get a different value per task. this is an error we have to fix in soopervisor. I think it should evaluate when running `soopervisor export`, store the value so that all tasks return the same value

can you open an issue? this is related to the existing problem we're discussing so we can tackle both at once :slightly_smiling_face:

no reason in particular :sweat_smile: we never encountered a problem. we use an internal package to manage releases: can you open an issue there? <https://github.com/ploomber/pkgmt> we can change it so it adds the missing `.0`

Compare to a baseline. I often find it's good to deploy early because I often learn a lot when the model is in production. Also the biggest win for me is fixing mislabeled data. And for some projects, I do need some creative feature engineering to solve it. Rarely do my projects fail succeed because of good HPO. It matters, I just find it matters less than other things and it's not worth the extra training time until the very end

Also... Best is sometimes hard to define. That's why it's helpful to deploy early because the end user doesn't care about f1 scores and you can have better precision / recall conversations when you have a real model running in production

Hey <!here> all, wondering if anyone is using Nix (<https://nixos.org/>) as their package manager and env for any Python work/ML related packages etc and what’s the feedback vs. other solutions? thoughts?

Link and Input are only available in the Python API. If you're using a pipeline.yaml, you can get something similar by registering passing the paths to the data as tasks parameters. if the files are not that big, you can use `resources_` so changes to the files trigger task execution: <https://docs.ploomber.io/en/latest/api/spec.html#id6>

re re-training: you can add an on_finish hook that evaluates the validation performance. if it isn't good enough, you can raises an EarlyStop error. This way the DAG will abort execution gracefully: there is an example here that uses the Python API but it works the same with a pipeline.yaml <https://docs.ploomber.io/en/latest/api/_modules/executors/ploomber.executors.Serial.html#ploomber-executors-serial>

re dbt: I don't have experience with dbt but sounds like a great idea for a blog post or an example. happy to help you!

sure. if you're using functions, just reference the other task:

```def my_task(product, upstream):
  one = upstream['one']
  another = upstream['another']```
if using scripts or notebooks, set a list in the upstream variable:

```upstream = ['one', 'another']```
does this solve the issue?

great. out of curiosity, did you look at some documentation pages? If you took a look at them and found them unclear let me know so we can fix them and add an explicit example

no I didnt look at them, I just added [] with you help :slightly_smiling_face:

ok cool. feel free to send any other questions!

Do you mean running it as a task? `ploomber task file.py`

Thank you Ido. Yes, but I was thinking in something more like a "rendered" script that can be solely run with Python. That is,  without the need to use Ploomber.

you can manually inject the values using `ploomber nb --inject`

great question! the output notebooks generated by NotebookRunner are uploaded to S3. If you want to aggregate results, you have a few options.

1. download the output notebooks from S3 and extract the notebooks using our sklearn-evaluation tool <https://sklearn-evaluation.readthedocs.io/en/latest/user_guide/NotebookCollection.html>
2. create a final task in your pipeline that aggregates the notebooks with sklearn-evaluation. then download the final notebook
3. add mlflow or any other experiment tracker to log the results. we have an example here: <https://github.com/ploomber/projects/tree/master/templates/mlflow>
does this help?

Hello!

You can run parametrized pipelines, in this case you can define the input parameters in a yaml file and then run the pipeline using the values in the file.

More about this: <https://docs.ploomber.io/en/latest/user-guide/parametrized.html>

Please note that you can also override values in this file by running the build command with a specified flag:

```ploomber build --force --env--some_param another_value```

you can execute bash scripts from ploomber (and run the scrapy command): <https://docs.ploomber.io/en/latest/user-guide/shell.html#shell-tasks>

alternatively, you can create a notebook and use the `%%bash`  magic to execute the commands:

```%%bash
scrapy ...```

Have a spider that extracts tables from a few pages. Execute it via the command
```scrapy crawl &lt;spider-id&gt; -o &lt;file-name&gt;.json```
the command above needs to be executed at the rool level of where the scrapy project was initialized

If your crawler is part of your pipeline, you can use the pipeline parametrization (even using a grid task) to run your crawlers with specified ids.

So far the best solution I’ve found is to create a separate module with functions A &amp; B, create tasks A &amp; B within both the training and inference pipelines with different parameters,  import the module functions within each task and apply different parameters. In this way, we ensure the training and inference tasks A&amp;B remain the same, but we can apply different parameters. The downside is that separate training and inference tasks A&amp;B need to be created.
Do any of you use a different approach in this type of scenario?

you have two options

1. since it's only two tasks that are shared, you could declared each one of them in the training and inference pipeline, then pass different parameters
2. if you want to stick to `import_tasks_from`, you can define placeholders in the parameters
```source: some-task
params:
  # define a placeholder
  some_param: '{{some_param}}'```
then, you could have `env.train.yaml` , and `env.inference.yaml`  and define the values there:

```some_param: some_value```
to <https://docs.ploomber.io/en/latest/user-guide/configuration.html#managing-multiple-pipelines|switch between> the two envs, you can set the environment variable before running the pipeline:

```export PLOOMBER_ENV_FILENAME=env.train.yaml &amp;&amp; ploomber build```
does this work?

Thanks for your quick reply Eduardo. This is very helpful!

no problem. if this doesn't work for you, let me know!