No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

ploomber is agnostic to what the source code contains, so taks can execute spark, dask or anything else. is this what you're thinking?

Right, one could define spark operations in a task and, although the task code would be interpreted locally, the computations would be delegated to the configured spark instance.

I'm interested in something more similar to <https://github.com/joblib/joblib-spark|joblib-spark>.

ah interesting. so this is using the spark cluster to run the code. can it run arbitrary python code?

I believe it's dataframe operations only. So, if there is a possible integration with ploomber, it won't fit ploomber's conception of an executor.

I'll look into some way of parameterizing a notebook or script to either run in pandas locally or distribute jobs to a config'd spark cluster.

we're working with the team at Fugue for an integration. their backend supports Spark, so I think once that's ready, then ploomber will support Spark :slightly_smiling_face: <https://github.com/fugue-project/fugue>

a similar thread:
Can we specify the interpreter?

```# src/ploomber/tasks/notebook.py
if self.source.language == 'python':
  interpreter = _python_bin()```
use case:
Connect to local interpreter (default) or remote interpreter. Databricks has a <https://databricks.com/blog/2019/12/03/jupyterlab-databricks-integration-bridge-local-and-remote-workflows.html|remote kernel integration> that allows local notebooks to be executed on Databricks. This is probably also something that could be parameterized at the task level, but it'd be nice to toggle for an entire DAG.

*note*: I'm (probably incorrectly) equivocating interpreter and kernel here, but I hope the intent is clear.

short answer: yes! we execute the notebooks using papermill and it's possible to choose a different kernel . let me dig a little bit to provide a more detailed answer

ok, so I read the article. looks like it's possible. you can customize this:

```from pathlib import Path
from ploomber import DAG
from ploomber.tasks import NotebookRunner
from ploomber.products import File
dag = DAG()
NotebookRunner(Path('nb.ipynb'), File('report.html'), dag=dag, papermill_params=dict(kernel_name='kernel-name'))
dag.build() ```
to list the kernels available, you can run `jupyter kernelspec list` , then you can substitute `kernel-name` for the kernel you want to use. I don't have access to a databricks cluster, but let me know if this works

hi, we have a cloud service to run pipelines in a scalable way: <https://docs.ploomber.io/en/latest/cloud/index.html> we charge for infra usage

I see, so currently there's no plans to gateway/paywall any features aside from compute/deployment?

nice <@U03SGJ7DKQD> - what client do you use to execute your queries?

cool. maybe not very applicable to you but I'd still love to hear your feedback. we're working on a sql client for jupyter and released a feature to simplify building large queries <https://jupysql.readthedocs.io/en/latest/compose.html>

In the grid documentation the example for parameterizing outputs doesn't include numbered suffixes

but when I use parameters in my product names

for instance `csv: "products/example-[[env]].csv"`

gets injected as `"csv": "products/example-prod-2.csv"`

yes that's expected and also weird :sweat_smile: before we introduced the `[[placeholder]]`   feature, we automatically numbered the files to prevent clashing names, now they're not strictly necessary but they help ensure each filename is different. I think we should add an option to turn it off - in the meantime we'll add a little note to the docs because I agree it's confusing, thanks for your feedback!

That looks pretty useful. Can/do functions that are not depending on each other run asynchronously?

if the functions are defined on a different file, then it works. if they're defined inline, there's some issue with the Python multiprocessing module. I need to investigate this more

Looks good! I've spent a little time attempting to do this with more generic functions. E.g., something like

```def ones() -&gt; pd.Series:
  . . . 

def plusone(ds: pd.Series) -&gt; pd.Series:
  return (
      ds
      .add(1)
      .rename(f"{ds.name}_plusone")
  )

def join(ds1: pd.Series, ds2: pd.Series) -&gt; pd.DataFrame:
  return ds1.to_frame().join(ds2)

def make_dag():
  return join(ones(), plusone(ones()))

dag = make_dag()```
I figure there's a decorator design that would allow such arbitrary python callables to be converted to ploomber tasks. I've played around with `PythonCallable` some, but haven't locked in a working pattern.

Additionally, it'd be useful to create dags that can accept data from memory.`input_data_parser` seems to allow this, but, again, I haven't landed on a working pattern. This way, common micro-pipelines could be imported from installable packages and used to transform data in memory, aiding rapid prototyping. As the project matures, the dag could be converted from an `InMemoryDAG` to a `DAG` and the results could be saved to disk.

Do `InMemoryDAG`s allow caching an incremental loading? Ephemeral storage of intermediate results, perhaps via an explicit checkpoint flag, could be useful when maturing from prototyping to prod.

&gt; Looks good! I've spent a little time attempting to do this with more generic functions. E.g., something like
are the arguments of these functions the upstream dependencies? for example `def join(ds1, ds2)` - are ds1 and ds2 expected to be tasks in the pipeline?

&gt;  Additionally, it'd be useful to create dags that can accept data from memory.`input_data_parser` seems to allow this, but, again, I haven't landed on a working pattern.
yes, `input_data_parser` is the way to go.

&gt;  This way, common micro-pipelines could be imported from installable packages and used to transform data in memory, aiding rapid prototyping.
i like this idea

&gt;  the dag could be converted from an `InMemoryDAG` to a `DAG` and the results could be saved to disk.
currently, you first need to build a DAG and then convert it into an InMemoryDAG, the other way around is not possible, but that's just implementation details, from the user perspective, we can make this transparent

&gt; Do `InMemoryDAG`s allow caching an incremental loading? Ephemeral storage of intermediate results, perhaps via an explicit checkpoint flag, could be useful when maturing from prototyping to prod.
it doesn't, since it passes outputs in-memory, it doesn't store anything. but you can get that with `DAG`.

I'll work on the example I shared earlier to add a few more features, like the data parser, dag -&gt; inmemorydag conversion etc. I'll share my updates here :slightly_smiling_face:

&gt; are the arguments of these functions the upstream dependencies? for example `def join(ds1, ds2)` - are ds1 and ds2 expected to be tasks in the pipeline?
I see them as naive functions. They could be used entirely without ploomber. One could pass any two data series or dataframes (or any `.join`able objects) to `join` and get a valid result. `ds1` and `ds2` could be outputs of upstream operations *or* they could be sideloaded from memory / disk. In the sideloading case, these objects would probably be passed through `input_data_parser` so they'd be outputs of an upstream task.

Thanks for all the other notes! This clarifies a few things. Looking forward to riffing on this more and seeing what you come up with.

I got something working last night :slightly_smiling_face: now the parallel execution works even when defining functions in the notebook, and no need to use `product` in the function signature. this requires some changes in Ploomber's internals but I can get something working next week. let me know what you think! <@U03RUC3NVDL> <@U03RJK306UA>

I'll check the docs. Perhaps we're not mentioning it explicitly. It looks at the source code, if it has changed it reruns. Since parameters are part of the source code, parameter changes will trigger a run 

yea that matches what I gathered from documentation for the ploomber build cli option

kind of out there but I assume there isn't a way to "expire" a query

for instance deciding that a query is only valid for 5 minutes or 24 hours

depending on how frequently you care about having the most recent results

interesting use case, haven't thought about it before. let me try something, I'll keep you posted

alright, so there are a couple options here:

are you running the pipeline on a schedule? if so, you can run it every 5 minutes  /24 hours with  `ploomber build --force` and it will run everything - even if it hasn't changed

if you're not scheduling, you could implement this logic with some Python code. This is a bit hacky but it'll work. <https://github.com/ploomber/projects/blob/6b7ec4a7d1e220c90cce9271a6195d2768609d58/cookbook/python-load/pipeline.py#L57|Here> we have an example where we load a pipeline into Python and overwrites  the task status. In this case, even if the task has changed, we skip it. in your case, you want a different thing. You can do something like:

```# if the task does not have any dependencies
 dag['your-sql-task']._exec_status = TaskStatus.WaitingExecution

# if it has dependencies
 dag['your-sql-task']._exec_status = TaskStatus.WaitingUpstream```
let me know if this works for you!

the final thing is a long term feature to prevent the hacky solution above. so let me know what you think

we could allow dynamic parameters in tasks. example:

```tasks:
  - source: script.sql
    params:
       my_param: my_params.compute_param```
then have a `my_params.py`

```def compute_param():
    return 'some-value'```
and pass  'some-value' to `script.sql` - this would allow you to define some custom logic, since this parameter will be part of the SQL query, then you could define something that changes the value every 5 mins / 24 hours. etc. let me know what  you think. we can add it to the backlog

this likely has nothing to do with ploomber. Are you sure you're using the same kernel?

&gt; after installing jupyter hub 
what exactly do you mean by this? Jupyterhub is likely not running on your local machine, so would be a different environment than running from "jupyter notebook"

earlier it was running, somehow i deployed the same on different VM and now it fails

sorry, I'm not sure. I'd just try re-installing ploomber, checking your `PATH` and `PYTHONPATH`, etc.

<@U03J22AV0LD> it’s possible you installed it, shut down/restarted the machine? AWS usually have a startup script it runs on your machine on boot. If the ploomber installation isn’t there the machine won’t recognize it. We’ll create a video soon on how to bootstrap Sagemaker (Which is the one you’re using I assume?)

i have installed all the required libraries and am able to execute the notebooks individually

when building the pipeline , they fail saying no module found

looks like the Python env running in your notebook and then one where ploomber is running is not the same.

can you run this?

```!  python -c 'import sys;print(sys.executable)'```
and in a Python cell:

```import sys
print(sys.executable)```
btw, you can also use the Python API to build your pipeline, it has nicer output:

```from ploomber.spec import DAGSpec

dag = DAGSpec('path/to/pipeline.yaml').to_dag()

dag.build(force=True)```

thanks for your help, i have started using python API

my observation, despite passing the parameter show_progress=True i do not see the progress bar

are you using the parallel executor? I saw a problem when printing progress

I see you're using the `debug=later` , awesome! let me know what you think

I added a progres bar in the parallel executor, it'll be out in the next release :slightly_smiling_face:

when building pipeline using python api , kills the whole  system

do you see an error when using the Python API? Does the error also happen if you use the serial executor?

i havent tried the serial executer, but when using parlallel it just kills the VM

oh weird. that'll make it challenging to debug. i'm glad you find a workaround. I'll take a look at the source code and see if I find anything that might be causing the issue. thanks for your feedback!

yep. scroll down a bit here: <https://docs.ploomber.io/en/latest/api/_modules/tasks/ploomber.tasks.NotebookRunner.html>

i have a huge pipeline running on production , when running the command report i only see 4 files yet to run, however when plotting the same it shows more than 4 to be executed

this is hapenning since i have upgraded ploomber

what happens if you change both commands to do `DAGSpec('path/to/pipeline.yaml').to_dag().status()`  and `DAGSpec('path/to/pipeline.yaml').to_dag().plot()` ? does it still show the inconsistency?

ok, and what happens when you build the pipeline? does it run the ones that the status or the plot say?

```Remaining: 'BONUS_LIFT_CALCULATOR-BETBOOTR', 'CUSTOMER_LIFECYCLE_MANAGEMENT-PREMIUM-BETBOOTR', 'PROMOTION&amp;CAMPAIGN ANALYTICS-BETBOOTR', and 'TOP CUSTOMER IDENTIFICATION CARD-BETBOOTR'```
above are the remaining notebooks to be executed , however the report and plot shows completely different notebooks

ok, can you share the full output of status? I see it only has four tasks but it should print one line per task

do you have any stable version so that i can put back my production loads

it's hard to tell what's the problem, and if downgrading will solve. if you recorded the version before upgrading, I'd say revert back to that version and see if the problem goes away

also, you might want to restart the session, and run status right after plot. ploomber looks at metadata files so if the metadata changed, the results won't look the same

My Observations on version 20
1. Its overburdening the system and starts killing the kernal despite having enough RAM

am on 0.15.2 , i still see inconsistency  between report and plot

Can you point me to the errors you're seeing in pipeline-2.html? I counted the "outated=False", and the others. and they match. I went through them individually and i couldn't find a mismatch

Has not been run. =9  , however in the graph i see more that 9

In the html you shared, the ones that say “False” are up to date. So they appear on green. Any other value is “outdated” hence in red. I don't see anything that hasn't been run in the html.

why do i get outdated when am using --force to run the pipeline

am sorry if am trying to understand it wrong

no worries. this is really helpful. it means our plot and reports are not clear enough, and that's on us :slightly_smiling_face:. if you do `ploomber build --force` that will run all tasks regardless of status. however; when creating the report/status/plot, it'll report the status (since those commands don't take the `--force` arg

how do i know if a task failed ?  from the graph and from the table

the run status is printed after you run `ploomber build`  for any task that fails, you'll see the traceback of each one

but once you run `ploomber build` ; you cannot recover the pass/fail status with plot/report

Hi!

1. This is currently unsupported. I forgot to document this so thanks for your feedback. we'll add it!
2. If your environment is called `something`You can create an `env.something.yaml` file. Soopervisor will pick that up and use it instead of the `env.yaml` by default. Let me know if this solves your issue!

Thank you for the answers!
Out of curiosity, do you have any intention of adding support to dag-level hooks in the future?

About the params, I meant something more like this:
```apiVersion: <http://argoproj.io/v1alpha1|argoproj.io/v1alpha1>
kind: Workflow
metadata:
  generateName: project-name-
spec:
  entrypoint: dag
  arguments:
    parameters:
    - name: param1
      value: value1
    - name: param2
      value: value2
  templates:
  - inputs:
      parameters:
      - name: task_name
    name: run-task
    script:
      command:
      - bash
      image: image-name
      source: |-
        ploomber task {{inputs.parameters.task_name}} --entry-point pipeline.yaml --force \
        --env--param1 {{workflow.parameters.param1}} --env--param2 {{workflow.parameters.param2}}
      volumeMounts: []
      workingDir: null
  - dag:
    ...```
basically being able to override parameters from the workflow.

Re dag-level hooks: it's now on our radar (we prioritize things that users bring up, so thanks!); but I can't promise a date. we're a small team :sweat_smile:

Re env: maybe there's something I'm missing here but you can achieve the same result with the solution I suggested

`env.yaml` contains your environment default values
`env.argo.yaml` contains the values you can use in Argo (you can use `import from` to get the values from `env.yaml` so this second file only has the values you need to override, <https://docs.ploomber.io/en/latest/user-guide/configuration.html#env-yaml-composition-dry|see here>)

then, if your soopervisor environment is called `argo`, then soopervisor will use `env.argo.yaml` as the environment, achieving the same effect as if you did `--env--param` inside the YAML. Does this work?

Re hooks: got it! Thank you for your effort!

Re env: I guess the use case is a little different, if I'm not missing something.

With the `env.argo.yaml`  I can specify parameters for any environment, but those parameters are also bundled within the generated image.
The use case I'm facing is instead generating more of a "template" - I have a pipeline which has some default parameters, but I can choose to run the workflow with new parameters without re-running soopervisor.
(That is, if I have understood correctly how env files work)

ah I got it now. So the problem is that the parameters as bundled with the image but you want to use the same image with different parameters. it's not supported currently

interesting, this is something I didn't think before. I'll open an issue so we keep track of it!

created an issue: feel free to add your feedback <https://github.com/ploomber/soopervisor/issues/108>

great point! we think data pipelines fit well in the procedural paradigm. when you introduce objects, you have a lot of share state between the tasks in your pipeline that can cause a lot of issues. of course, we won't stop you if you decide to build your own class-based DSL on top of ploomber, but we think the procedural one works well :slightly_smiling_face:

however, I think objects make sense inside tasks in a pipeline. for example, to represent things that *do* need state like models that have learned parameters after they're trained

I totally agree that using mutable state in a class is a bad idea for a data pipeline.
But on the other hand I think using immutable state in a class can make the code cleaner.
The data that are immutable and used by different pipeline stages can be accessed from the class.
A typical example is the configuration of a specific task. Without class I have to pass the context to every function that depends on it.  For example, which Slurm cluster to submit the job to, etc.

And logger is another example. But I think that's not a big deal compare with other other awesome feature prodvided by ploomber.

I agree with your configuration argument. Under the hood, Ploomber represents your pipeline as an object precisely for that. But it manages the state for you. For example, if you use the S3 client, It'll reuse the same client but you don't have to touch it because Ploomber will automatically upload artifacts. Still, you have a point, we actually have an open issue about how to share state; but I haven't found the right abstraction to do it, so if you have suggestions, let me know!

you're right! it's a problem on our end. papermill was an optional dependency but now it's mandatory, so we should move it in the setup.py. PRs welcome!

I would like the opportunity to try to submit this PR. Seems like is straight forward. I guess just changing it to REQUIRES should work. What do you think?

yep. another thing to ensure this wont' happen is to update the CI configuration file: instead of simply installing dev dependencies, do:

```pip install .
python -c "import soorgeon"
pip install ".[dev]"```
this will ensure that if we only install mandatory dependencies, `import soorgeon` works without problems

because right now, we install dev dependencies and then test. that's why we didn't notice this error

edit here: <https://github.com/ploomber/soorgeon/blob/cb16dc51f326cf21486bff86dfec03b4d1f5c2d7/.github/workflows/ci.yaml#L31>

Oh, thanks. I will try to submit a PR with that.

Done. Let me know if I did something wrong. <https://github.com/ploomber/soorgeon/pull/86>

<@U0164GZJ5EK> would it be a good idea for me to work this week on <https://github.com/ploomber/soorgeon/issues/84|apply black before refactoring> ?  Maybe I could figure it out.

<@U0164GZJ5EK> I posted how got it working <https://github.com/ploomber/soorgeon/issues/84|in the issue> but hope u can take a look at it before submitting a PR because I might be missing something.

thanks! I'll take a look in the next few days

thanks for your feedback! yeah there might be a few things that we should do to ensure soorgeon's test work on windows, but we haven't thought about doing that. if you want to give it a try, feel free to open a PR!

you can achieve this by converting your file client into a placeholder:

```clients:
  File: '{{my_client}}'```
then, in your env.prod.yaml:

```my_client: clients.s3_client```
and your env.yaml:

```my_client: none```
then you can switch between these two with an env variable: <https://docs.ploomber.io/en/latest/user-guide/configuration.html#configuration-dev-prod>

Thanks so generally a 'none' will be treated as a null value, since all placeholders require to be strings I wasnt sure because i cant inject an actual null yaml value it always becomes 'null'

do you recommend reusing those clients in tasks or should they be kept only for pipeline internal purposes and init separate client for work that the task needs to do that includes IO. For example my task downloads a bunch of S3 files stiches them together and then has 1 product file.

good point. I think it's ok to re-use them. since the pipeline client already has the config, it'll make it simpler to embed it into your code

<@U0164GZJ5EK> it doesn't seem to work. Passing a value of none triggers `TypeError: Expected dotted path spec to be a str or Mapping, got None (type: NoneType)`

``` File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/cli/parsers.py", line 495, in load_dag_from_entry_point_and_parser
    dag, args = _process_file_dir_or_glob(parser)
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/cli/parsers.py", line 447, in _process_file_dir_or_glob
    dag = DAGSpec(dagspec_arg, env=env).to_dag()
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/spec/dagspec.py", line 484, in to_dag
    dag = self._to_dag()
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/spec/dagspec.py", line 521, in _to_dag
    dps = dotted_path.DottedPath(dotted_path_spec,
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/util/dotted_path.py", line 52, in __init__
    self._spec = DottedPathSpecModel.from_spec(dotted_path, strict=strict)
  File "/home/ec2-user/anaconda3/envs/44/lib/python3.10/site-packages/ploomber/util/dotted_path.py", line 444, in from_spec```

are you passing the client to each task?

```tasks:
  - source ...
    client: ...```
or do you have a pipeline-level client?

```clients:
  File: clients.get_s3_client```
I think the task-level won't work but the pipeline one will

It's a pipeline level and the following error is when the value is set to none or "none" as advised to run local filesystem in the dev env. Seems like it iterates over the set of clients and expects a string but receives null value as defined by the none which explodes.

ah, I think I might have introduced this in a recent update. can you open an issue?

<https://github.com/ploomber/ploomber/issues/1015>

does this happen while running `ploomber build`? it has happened to me before, often when the notebook uses a lot of memory

you   may wanna check our profiling tool to see if that's an issue: <https://github.com/ploomber/ploomber-engine#notebook-profiling-cpu-gpu-ram>

I am thinking it is a memory issue as well. The ploomber build command works perfectly with 0-80% of the dataset, but when I get in the 80-100% use of the dataset I get this error.

if you're using the parallel executor, you may wanna switch to the serial one to reduce memory usage. if you're already using the serial executor, probably getting a larger machine is the easiest option. Or split the notebook into two.

are you using pandas? memory usage tends to blow up with data frames

Thanks! I will take a look at this. That is correct this is happening using the ploomber build command. The data scientist is using pandas.

This is the pipeline.yaml file:
```tasks:
  - source: scripts/get_feat_eng.py
    product:
      nb: 'products/{{out}}/sample={{sample}}/get_feat_eng.ipynb'
    params:
      sample: '{{sample}}'

  - source: scripts/train_test_split.py
    product:
      nb: 'products/{{out}}/sample={{sample}}/train_test_split.ipynb'
 
  - source: scripts/fit_predict.py
    product:
      nb: 'products/{{out}}/sample={{sample}}/fit_predict.ipynb'```

ok so this is using the serial executor. my advice is to see if you can optimize the script that's failing, there are a few tricks to optimize memory: <https://stackoverflow.com/q/39100971/709975>

one quick thing would be to split the failing notebook into two, this may fix the issue since once "task-part-1" finishes executing, ploomber kills the kernel, freeing up the memory, then you can proceed to execute "task-part-2", maybe run some profiling.

if you have more questions, feel free to ask

Thanks a ton <@U0164GZJ5EK>! Knowing that the kernel is killed and frees ups memory after each task in the pipeline.yaml file helps a lot. The data scientists was doing the bulk of the workload in the initial script and she was planning on splitting that task later to different components later in her work flow. I will just suggest for her to do that now vs later.

<@U0164GZJ5EK> just as an update. Splitting the task that was failing to 2 parts did the trick. Thanks again!

yes, you can! you can find the Dockerfile in `{env-name}/Dockerfile` , you can edit manually and then soopervisor export will use that one

great question! we have an open issue that will <https://github.com/ploomber/ploomber/issues/602|simplify this use case> - but haven't finished working on it yet!

I'd suggest adding one <https://docs.ploomber.io/en/latest/user-guide/parametrized.html|parameter> to pass the columns you want to train on:

```# pipeline.yaml
tasks:
  - source: train.ipynb
    product:
      nb: 'output/{{experiment_name}}/train.ipynb'
    params:
      column_to_delete: '{{column_to_delete}}'```
then, create an `env.yaml`:

```experiment_name: default_experiment
column_to_delete: null```
then implement the logic that drops a column based on the `column_to_delete`
value, and run your pipeline with:

```ploomber build```
to run an experiment with a column dropped:

```ploomber build --env--experiment_name my-experiment --env-column_to_delete some-column```
then you'll have `output/default/train.ipynb`  and `output/my-experiment/train.ipynb`  and you can compare them!

if you have more than one task in your pipeline (e.g. the tasks that generate the training set, you can share those files), this will allow you to cache results so next time you run an experiment, you don't have to run all tasks

```# pipeline.yaml
tasks:
  - source: prepare.ipynb
    product:
      # note that we don't use "experiment_name" here!
      nb: output/prepare.ipynb
      data: output/train.csv

  - source: train.ipynb
    product:
      nb: 'output/{{experiment_name}}/train.ipynb'
    params:
      column_to_delete: '{{column_to_delete}}'```
let me know if this helps!

Thanks for the detailed explanation! In fact this is what I've done for now. I created a 'v1' with the current columns and a 'v2' with the additional  column by using env.yaml and params. However, I would like to compare the results in a last task that combines all (or given versions). This seems to be exactly what the issue is about that you linked.

The problem is that currently I can not easily cascade the params to my other notebooks / scripts in the pipeline. And because the selection of the columns comes very early in the pipeline I have to run everything multiple times. Of course this is not possible to avoid but I lack the possibility to easily combine the results later on. Within a given pipeline I'm always stuck to the current parameter.

Ideally I want to be able to change (or add) any task in my pipeline (e.g. add a feature) and compare the results to see how this change affects the results. If I understand correctly this will be possible with <https://github.com/ploomber/ploomber/issues/602> but currently it is not possible to do this within the pipeline, i.e. you have to manually look at the different pipeline results or create a comparison script outside the pipelines. Is that correct?

&gt;  However, I would like to compare the results in a last task that combines all (or given versions). This seems to be exactly what the issue is about that you linked.
ah, i thought you wanted one evaluation task per task generated by grid. if you want to evaluate all models at once, there's another thing you can do. you can use grid and set the next task with a placeholder:

```tasks:
  - source: ...
    grid: ...
    name: train- # since this is a grid, it'll generate train-0, train-1,...

  # make this task depend on all fit- tasks
  # by setting a placeholder as the upstreamm:
  # upstream = ["train-*"]
  - source: evaluate.ipynb
    product: ...```
&gt;  The problem is that currently I can not easily cascade the params to my other notebooks / scripts in the pipeline.
do you need to access the grid parameters in later tasks? one way would be to have the train.ipynb store its parameters in a `parameter.json` and register it as a product, then load them in the next stage. Alternatively, you can use <https://sklearn-evaluation.readthedocs.io/en/latest/user_guide/NotebookCollection.html|our notebook introspector> to extract values, charts, table from output  cells

I think both of these things combined would get you what you want,  but let me know if this is not what you want to implement

btw, a final alternative is to use the Python API directly, which is a lot more flexible for this kind of dynamic things, the <https://sklearn-evaluation.readthedocs.io/en/latest/user_guide/NotebookCollection.html|notebook introspector> has a complete example

Thanks for the hint with notebook introspector! This is also going into the direction I was looking for. The parameters cascading I can probably accomplish then also via the DAG as well. I will try and see if I can achieve it :slightly_smiling_face:

great, feel free to post other questions if you need help! I'd love to see that notebook pipeline up and running!

hey, congrats on starting the phd program!

currently, the <https://docs.ploomber.io/en/latest/cookbook/debugging.html#post-mortem-debugging-debug-later|debugger> only supports Python, but you can get a similar effect on R. If your R sources are scripts, first inject the cell, then you can use <https://adv-r.hadley.nz/debugging.html|R's debugger> - if your sources are not `.R` files (e.g. `.Rmd`), you can inject the cell, convert to R (something like this will work, although I'm unsure about the exact argument name `jupytext some-file.Rmd --to some-file.R` ), then use R's debugger

that is, call the script from R (outside of Ploomber)?

yes! calling the R script outside of ploomber

Hopefully/preferably in a language agnostic way so we can drop into the R debugger 

thanks for your feedback! this looks like a bug, can you open an issue?

just saw this sorry Eduardo I will see if it already was picked up.

Right, we do not copy the entire repo, as it can get big. when working on soopervisor, I forgot about those git placeholders :sweat_smile: can you open an issue?

<https://github.com/ploomber/soopervisor/issues/114>

these errors are pretty hard to debug, unfortunately.  there is a tool to debug github actions (<https://github.com/nektos/act>) but i dont think it supports windows. sometimes the CI just fails randomly, so might worth re-running it

Hi all! How often do you have version incompatibilities? and how do you approach them?

made-up example: you run `pip install pandas` and then try do to:

```import pandas as pd

pd.read_csv("data.csv")```
And you get a `AttributeError: module 'pandas' has no attribute 'read_csv'` error. After some digging, you might find a version mismatch and either determine what's the equivalent function/class in the version you are running or upgrade/downgrade your installation