This message was deleted.

The step1 -> step2 -> step3… structure is pretty common and the getting started tutorial can help you there. You can also find the parallelization guide as well, this sounds like a perfect use case for grid, using multiple parameters in parallel.

hey @Klaus: as Ido mentioned, the grid feature can help here. but looks like you want a grid that spans multiple notebooks?https://github.com/ploomber/ploomber/issues/602 - let me know if the github issue describes what you want to do

To be more precise: The parameters are stored in a pd.dataframe with n rows (scenarios) and m columns (parameters). The notebook steps should run sequentially (step 1 -> step 2 -> step ...) as step 2 needs the result of step 1 and so on. Each step should run all scenarios in parallel as the scenarios are independent of each other. And I use Windows and Ubuntu for development. But right now, Windows is more important.

so there are two ways to achieve paralellism. either you create one notebook (one task in your ploomber pipeline) and implement parallelism inside the task (e.g., by using joblib or similar). or you create N ploomber tasks (each one with one set of parameters) and then execute your pipeline with Ploomber's parallel executor. If you go with the latter case, you might be better off using the Python API, as it provides more flexibility for dynamically building pipelines: https://github.com/ploomber/projects/tree/master/python-api-examples

I thought my use case was a standard use case for ploomber. But. it seems that my use case is not a standard use case for ploomber and that it may be more convenient to use the standard multiprossing library. Even if i have would have to convert the notebooks into python scripts. I'm aware that the full benefits of Ploomber would not be available. Am I wrong? I wish you could answer yes 🙂 and point me to a tutorial, a discussion thread, or give me some more details for my use case.

the simplest advice I can think of is to use ploomber-engine with the multiprocessing library, this way you don't need to convert the notebooks (we haven't tested it running with the multiprocessing library so let me know if you encounter any issues) https://engine.ploomber.io/en/latest/user-guide/running.html engine is great because it allows you to run notebooks and then extract the results from them: https://engine.ploomber.io/en/latest/user-guide/testing/integration.html so you could use it to build a pipeline with A.ipynb -> B.ipynb -> C.ipynb and then use the multiprocessing library to run this pipeline with X combinations of parameters. should be fairly easy to implement

@Klaus were you able to get it working? Are you up for a PR? (we think the community can benefit from documenting it).