No data migration. Ploomber Cloud runs on AWS, and can integrate with your existing infrastructure such as S3, GCS, RedShift and Athena.

Ploomber

you can enable multiprocessing for running pipeline tasks in parallel in your pipeline.yaml
```executor: parallel```
if you wish to use multiprocessing.pool inside a task that's also possible

image.png

ok thank you, a few of my files have multiprocessing in them.

let's say I have a pipeline that looks like this, how could I get each red box to be able to execute at the same time?
say, allow groups (1,2) (3,4,5,6) (7,8) to run in parallel

The parallel executor will do that for you. However, there is no way to enforce that certain tasks run at the same time, the parallel executor will create a pool of processes and schedule new tasks as others finish 

For example. It task 1 takes an hour to finish but 2,6,8 are pretty fast, the pool still has to wait for 1 before it can continue 